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ABSTRACT 

The four papers in this collection were drawn from a 
symposium convened to review the state of the art in speaking and 
listening assessment at all educational levels and to suggest 
strategies for addressing ^e technical issues, particularly those 
relating tc test bias and the use of rating scales, that speaking and 
listening assesscrs often f*ce<> The first paper presents resultfe of a 
national survey of elementary and secondary speaking and listening" 
assessment procedures that was sponsored by the Speech Communication 
Association- Various sections of the paper focus on s^ate legislative 
activity, specific skills tested, standards, and testing procedures- 
The paper reports -the findings of the survey on a state-by- state 
basis- The second paper deals with the current assessment of speaking 
.and listening skills at the college level and concentrates cn the 
development and evaluation df the Communication Competence Assessment 
Instrument- The third paper reviews potential sources cf bias in 
^assessments-including stimulus conditions, response mode, rating 
\procedures, end use of test results--and suggests strategies for 
dealing with eaclu The fourth paper identifies several issues related 
to the use of rating scales in evaluating oral communication 
proficiency— including scale development, psychometric evaluation, 
and the speech type assessed--and suggests new directions fcr 
research* (PL) 
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This collection of papers was prepared for presentation as a symposium at 
the Annual Meeting of the American Educational Research Association, Los 
Angeles, 1981. 

The material presented .herein was developed by the Clearinghouse for 
Applied Performance Testing (CAPT) under grant #400*80-0105 .with the 
National Institute of Education (NXE) of the Department of Education* 
The opinions expressed in this publication do not necessarily reflect the 
position of NIE and no official endorsement by the Institute should be 
Inferred* 



INTRODUCTION 



The^aasessment of speaking and listening -skills is booming a high 
priority nationwide Numerous state level assessment programs consider 
communication skills important targets for basic skills assessment, and 
several assessments focus specifically on the assessment of speaking and 
listening. Zn addition, the Speech Communication Association has 
assembled a national task force to document research and development in 
speaking and listening assessment and to promote sound assessment 
procedures* * 

This symposium was convened to provide a relevant, up-to-date picture 
of current issues, recent advances and needed research relating to the 
measurement of speaking and listening skills. Specifically, the purposes 
of thin symposium were to (1) reviey the state of the art in speaking and 
listening assessment at the elementary, secondary and poatsecondary 
levels, and (2) suggest strategies for addressing technical'* 
issues—particularly those relating to test "bias and the use of rating 
scales— faced by speaking and listening assessors. 

o Speaking and listening assessment is an important topic in 
educational, in part because it represents an opportunity to use : 
performance assessment, a viable alternative to traditional paper and 
pencil assessment* Performance assessment relies on the direct 
observation and evaluation of behavior by qualified judges under real or 
simulated circumstances* This form 0 of assessment is rapidly gaining 
acceptance for several reasons: 

1* There is growing interest in assessment methods in which 
results can be generalized to behavior outside school* 
High fidelity performance tests, such as those designed to 
measure speaking and listening skills, maximize the . 
external validity of the assessment. ^ — — — _ 

2* Seme evidenoe suggests that performance measures can 
minimize the adverse impact of testing on minorities. 
Speaking and listening assessment has a high potential for 
bias due to the heavy cultural loading of oral 
communication* However, carefully developed assessment 
materials and approaches can greatly reduce problems of 
bias* 

3* A growing ntjmber of business, vocational and industrial 
training programs, as well as professional licensing and 
certification programs provide new settings for the 
development and uee of performance assessment* 

The papers included in the symposium present a variety of 
perspectives, on performance assessment in the evaluation of speaking and 
listening skills* In the first paper, Philip Backlund, of Central 
Washington University, and his associates present results of a national 
survey of elementary and secondary speaking and listening assessment 
procedures sponsored by the Speech Communication Association. The survey 
focused on state legislative activity, specific skills tested, standards 
and testing procedures* Results of the survey are reported on a 
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slate-by-state basis. 1 In the second paper, Rebecca Rubin, of the 
University of Wisconsin-Par kside, deals with the current assessment of 
speaking and' listening "skills' at the college level. Rubin's paper . 
concentrates cn the development and evaluation of the Communication 
Competence Assessment Instrument. 

Richard Stiggins of the Clearinghouse for Applied Performance ~' 
Testing, northwest Regional Educational Laboratory, reviews potential 
sources of bias in speaking and listening assessment and suggests 
strategies for dealing with each. Specific sources of bias are • 
identified in stimulus conditions, response code, rating procedures and 
use of test results. Donald Rubin of the University of Georgia then 
identifies several important issues related to the use of rating scales 
in evaluating oral communication proficiency. In particular, Rubin 
covers scale devel o pm en t, psychometric evaluation and the type of speech 
assessed, and Suggests new directions for research. 

She presentation at the Annual Meeting of AERA included a fifth 
paper, "Referential Communication Activities: A Tool for Assessing 
Speaking and Listening in the Classroom; • by W. Patrick Dickson and 
Janice Patterson of the University of Wisconsin-Madison. Since the 
authors will be publishing their study in a research journal, they 
elected not to include it in this publication. 
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ABSTRACT 



The Speech Communicatica Association , through its task .force on 
aszesuaent and testing, has been active in monitoring states 9 progress in* 
the development of kssessaent programs in speaking and listening* A 
subcommittee of the task £orce was given the task of surveying each 
state* s current, practices and plans for the assessment of speaking and 
listening skills at the elementary and secondary levels* This paper 
summarizes the survey in three sections* The first section covers the 
objective of the survey and the* survey procedures* The objective was to 
gather information on legislative action, skills tested, setting skill 
level standards and testing procedures* The second section summarizes 
findings* The third section offers some critical comments concerning the 
findings* 
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A NATIONAL CURVET OP STATE PRACTICES IN 
SPEAKING AND LISTENING SKILL ASSESSMENT 



The description of the basic skills in the Elementary and Secondary 
Education Act (Title II) as 'reading and mathematics and aff active 
communication, both written and oralr* raflacts on expanded view ofr what 
is basic in education. Tb^ primary area of -expansion is oral 
communication. For decades, research and curriculum materials 
development in oral cbsmunication lagged significantly behind the work * 

' done in written pommuni cation. This is beginning to chang*. The • 
expansion of basic skills to include oral communication 'has led many - 
states and' school -districts to conduct research and develop instructional 
materials in oral communication* The Speech Communication Association" 
has attempted and is attempting, through a variety of means, to advise, 
inform, and coordinate these various efforts* 

Qoe such eff Orjt is being made through; the Task Force on Assessment 

^and Tatting. *ln* 1975 ,\ the Educational Policies Board. of the Speech 
Communication Association established thm Task Force, to facilitate the * 
flow of information on assessment of cdsmunication skills* from members of 
the speech communication profession to federal, state, and local 
educational leaders* 

The Task Force in turn was organised into three committees. One of 
these committees is called the "Committee on Current Assessment 
Practices, 1-12. ■ The purposes of this committee are to 1) ponitor 
assessment practices on a stata-by-state, and, when appropriate, local 
district basis; 2). gather and disseminate systematic data describing 
current practices; and 3) maintain and enlarge the Task Force file on 
resource personnel and assessment materials* This paper reports, the 
committee's activities and findings as of March 15, 19il.. , 

The report is organised into five sections: A) purposes and 
procedures, B) the Federal perspective on basic skills and oral 
communication, C) a sumaary of the findings, D) conclusions based on data 
gathered, and B) appendices that present a state- by-state description of 
activities in oral communication assessment* 



Purposes and Procedures 

The Committee believes this report serves three major purposes. 
First, it provides speech communication professionals and others who 
share their interest with information on state level assessment of oral 
communication* Second, it provides information on how each state is 
responding to basic skills legislation regarding speaking and listening. 
Third, it serves as an information clearinghouse to federal, state, and 
local school governing agencies concerning oral communication procedures 
and plans for the governing agencies* The Committee does not view this 
report as a final product*. Laws, regulations, and learning objectives 
are changing rapidly* The Committee will continue to monitor these 
activities and report their changes as new information is obtained* In 
keeping with this, the Conmittee would appreciate learning of any 
ommissions or inaccuracies the reader may note in this report* 
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The Committee oa Current Practices was created to cai^ry out these 
purposes* The Committee consists of Chairman Phil Backlund, Central 
Washington University? Dwayne VanRheeneh, University of Haine? Mike - 
Moore, Indiana University; Arlie Hullir Parks, Mansfield State College? 
and Jasies Booth, Murray State University. . 

To obtain the information, the cm— It tee conducted a telephone 
survey* We talked with either the Basic Skills Coordinator, the Language 
Arts Coordinator, and/or the Director of Testing and Evaluation in each 
state*' The following Questions wer* askedi 1) Bow is your state 
responding to the ba<lc. skills legislation in regard to speaking and 
listening? 2) pees your state have (or* is it developing) statewide 
learning objectives for speakihg and listening? 3) Are you currently 
assessing speaking and/or listening skills, or do you have plans for such 
-an assessment? 4) v if so, what instruments are used? 5) Which specific 
skills are assessed, and at what grade level? 6) How are the results 
used? 

'The ocamiittee presented its preliminary findings at the annual Speech 
Communication Association convention in November, 1980. The committee 
will continue to review and update the report as new data comes in. 



An Overview of Federal Activity in Basic Skills 

The impetus for much of the interest in basic skills dame from 
federal legislation enacted in 1978. Public Law 95-561 amended the 
Elementary - and Secondary Education Act of 1965 to include, (among other 
items) £ definition of basic skills that includes reading, mathematics, 
or&^obumunication, and written communication. This marked the first 
time that oral communication *had been so included in federal 
legislation* Title II of this Xatf authorized appropriations to fund the 
Basic Skills Improvement, Program. To 'provide an example of the type of 
programs funded under this legislation, here is a' summary of the 
appropriations* for . fiscal year 1980. 

Under Part A of the legislation, three kinds of demonstrations were 
authorized for -the improvement of the basic skills of both school 
children and youth and adults: In-School (Section 205), Out-of -School 
(Section, 208)", and Parent; Participation (Sections 206). Total funding 
for these demonstrations was. $15, 149 ,78 9. In addition, grants apd 
contracts' were* authorized under Pakrt A for the provision of technical 
assistance and for dissemination of . -information about the basic skills. 
Total cost of these dontracts 'was $2,781*666. Under Part B, grant *ere 
authorized for the support of two. effort! by state education agencies: . 
under Section. 222, states may support demonstration projects in 
elementary and v secondary schools, and under Section 224 they may support 
staff development in the basic skills ^at both the state and local 
levels. Appropriations; for. these, state grants , tbtal' $7,749,169. Part C 
of the Basic Skills legislation authorized two special programs — one for 
^improvement of reading through distribution of materials for reading 
motivation (RI^-Reading is Punrtamehtal) and the other in' mathematics. 
Appropriations for the former., yere $6,500,000 and for t#e iatter $586,252 
(U.Sw Dept. OfvBd., 1980). The U.S. Department of Education NEWS . 
•reported that. 156 grants and 15 contracts bhd been awarded under this 
•program as of January 30, 1981 (NEWS , 1981). 4 



x It is important to note, however r thAt^this legislation does not 
require states bo develop programs "in each of the basic skills areas* 
Bach state develops its own plans, sets its own priorities,' 4 and 
distributes grant funds as it sees fit* Thus, if a state chooses not to 
focus on any one of the basic skill areas (such as oral communication) , 
the Basic Skills Office of the Department of Education has no power to 
require the state to do otherwise* The Basic Skills Improvement Program 
is set up to assist, not to mandate program content (Broom, 1981) • 

President Reagan's economic policies have further impact on this „ 
program* The President is moving to change many formula grants into 
block grants* For education this implies, that individual states will 
have further control over their educational programs. Instead of 
receiving a number of grants for individual programs (such as basic 
skills) r the states will receive a block of money to distribute as they 
see fit* It is 'anticipated that this change will be implemented by 
October 1, .1981* In spite of funding changes, however, the Basic Skills 
Improvement Program of the U.S. Department of Education expects to 
provide strong support and assistance "services for the individual states 
in their efforts to develop quality basic skills programs* 



Findings Summary 

Bach of the states surveyed can be placed into one of five 
categories* The first includes those states that have no curriculum or 
assessment programs in speaking axiH/ot listening and have no plans of 
implementing any audi programs* 

The second category has expressed an intention to develop either 
curriculum or assessment programs in speaking and/or listening, but have 
not yet done fo* The third category includes states that tiave identified , 
speaking and/or listening skills and are developing statewide procedures 
to assess these skills* The fourth category includes states that have 
identified skills in speaking, and/or listening and are developing 
curriculum materials for teaching these skills but have no assessment 
procedure* The fifth category includes states that have identified 
skills and have developed (or are developing) both curriculum and 
assessment procedures for these skills* 

The first category includes nineteen states* These states are 
Alabama, Alaska, Arkansas, Colorado, Florida, Idaho, Kansas, Kentucky, 
Maryland, Mississippi, New Mexico, New York, North Carolina, North 
Dakota, Oklahoma, Tennessee, Washington, Nest Virginia and Wyoming* 

The second category includes fourteen states* These states are 
California, Connecticut, Delaware, Indiana, Iowa, Maine, Montana, 
Nebraska, Nevada, New Jersey, Ohio, Oregon, Rhode Island arid Wisconsin* 

The third category includes four states* These includes Hawaii is 
in the process of developing a speaking skill rating scale for use 
statewide in grades 3, .6, 8, 10 and 12. Indiana (Gary Community School 
District) has developed an. oral, proficiency examination as part of - 
'graduation requirements for its high school students* Massachusetts has 
identified fourteen speaking and eleven listening skills; testing of 
these skills has been contracted for with the National Assessment of x 
Educational Progress* New Hampshire assessed a sample of 5th, 9th, and 
12th grade students statewide in listening skills in October 1980* 
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\ The fourth category includes eleven states* Arizona has developed 
122 listening/reading' skills and 120 speaking/writing skills broken down 
by grade level* Georgia has both high school graduation requirements in 
speaking and listening (local authorities 3 are responsible for testing) 
and comprehensive Language Arts Guide which details resource and 
curriculum materials. Illinois is developing a list of specif ic oral 
communication skills for use in planning curriculum objectives for K-12. 
Louisiana has identified twenty-three oral -communication skills as part 
of their writing a sses s m ent program. Minnesota has a highly developed 
statewide curriculum support plan for all the basic skill areas- 
including speaking and listening. Missouri -(Independence School 
District) has developed specific speaking and listening competencies 
together with suggested curriculum activities designed to teach these 
competencies, north Carolina has developed a detailed list of speaking 
and listening competencies together with performance indicators. South 
Carolina has developed minimum standards for statewide educational 
objectives in reading, writing anil mathematics. Speaking and listening 
objective? are a part of reading and writing. South Dakota has published 
a well developed curriculum guide in speaking and listening to be used by 
local^districts in planning language arts programs'. Texas has developed 
curriculum objectives in speaking and listening for K-12. Virginia is 
preparing a set of objectives and assessment plans for all basic skills 4 
including speaking and listening. 

The fifth and final category includes four states. Michigan is 
developing a program of speaking and listening skills, curriculum 
strategies -to teach those skills, and assessment procedures tied directly 
to- the skills. Pennsylvania is developing a set of classroom assessment 
^instruments and curriculum materials designed to help the classroom 
teacher identify areas for instructional development in speaking skills* 
.Utah has a well developed statewide program combining basic skills 
(including speaking and listening) with life skills. Vermont has, a 
statewide program in speaking and listening skill assessment in place and 
reports statewide test results yearly. „ 

-The Speech Communication Association has published detailed state by 
state summaries. SCA also has copies ^of many state basic skill plans, 
>curriculum guides, and assessment procedures. Interested readers are 
urged to contact SCA for further information. 



Conclusions 

T * « 

It is not the purpose of this report to critically evaluate any state 
program* We can, however, make some observations and recommendations 
concerning state activities in speaking and listening. 

1) There is a great deal of interest and activity in oral 

communication, yet there is little available information. Many 
state basic Skill directors are highly interested in receiving 
pore information on oral communication. 'Professionals in speech 
communication and in education can provide' a valuable service by 
contacting their basic skills directors and local school 
authorities to offer assistance* 
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—2) While- many states have program in oral communication, many do 
pot* These states naad to ba sboim that programs in oral 
communication arm both dasirabla and feasible. 

3) Further coordination between state programs would ba useful. We 
discover ad some * reinventing the wheal" in our survey — that is, 
states developing similar programs without knowing what other 
states were doing, we can help provide audi coordination* 

4) It may be useful to urge more consistency in the various 
definitions of oral communication skills and assessment 
procedures. Current lack of consistency seems to be causing some 
coordination problems. It may be useful to urge states to more 
closely coordinate the identification of N these skills, perhaps 
using the model presented by the Speech Communication Association 
in its "Standards for Effective Oral Communication Programs," 

c jr. 

5) More research is needed in oral communication at the elementary 
and secondary levels. Specific areas' include effectiveness of 
instructional programs in oral communication, relationship of 
oral communication skills to written communication skills, and 
curriculum material development* 



In summary, oral communication is receiving much more attention now 
than only two or three years ago. Many states are including (or planning 
to include) speaking and listening skills in curriculum and assessment 
programs at the elementary and secondary levels. Despite significant 
progress, however, much work must be done before oral communication 
curricula and assessment methods are as fully developed as the materials 
in the other basic skill areas. As professionals interested in oral 
communication, we vi w that development as our overall goal. 



Results of the State Speaking fc Listening Survey * 
ALASKA: Nothing happening at this time. NO plans. 

Contact : 

Darby Anderson, Director 
Right to Read Program 
Department of Education 
Pouch F - Alaska »Off ice Building 
Juneau, AK 99811 

ALABAMA: ' Nothing currently in place or planned for speaking and .. 

listening. No definite response to the Basic Skills 
Legislation as yet. , ., 



"Note: This is an abbreviated summary of the original survey 



Contact: 

Mr. Bill Malone 

Edocational Specialist 

State Department of Education 

State Off ice Building, Room 402 

Montgomery, AL 36130 

(205) 852-3220 

ABS 15-102*26 'Suggested Guidelines for the 
Development and Implementation of a Continuous Uniform 
Evaluation System" appears to be their response to 
Title II* Speaking and listening are not addressed. 

Contact : 

Mr. Steve Stephens 

Research and Statistical Analyst, 

State Department of Education / 

1535 West Jefferson 

Phoenix, AZ 85007 

(602) 255-4888 

nothing is happening at this point in time concerning 
speaking and listening. No apparent plans. 

Contact : 

Or. Constance S. Dardin 
Coordinator, Student Assessment 
Department of Education 
ArchFord Education Building 
Lit/tie Rock, AR 72201 
(5dl) 371-1461 



No statewide testing program in speaking and 
listening. Local school districts have obligations 
for basic skill improvement programs. 

Contact ; 
Bill Burson 

California Department of Education 
721 Capitol Kail 
Sacramento, CA 95814 

Has developed a comprehensive program designed to 
train local agencies and personnel in identifying 
their own needs or basic skilis and developing 
corresponding programs. Thus, there are no state 
guidelines on the curriculum or assessment standards 
for any of the basic skill areas. Program is called 
•Colorado Comprehensive State Basic Skills Plan 19 and 
was legislated by Public Law 95-561. 
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COLORADO* (continued) 



Contact! 

Mr. Robert Bwy, Supervisor 
TitSi II, Basic Skills 
Colorado Department of Education 
201 E. Colfax 
Denver, CO 80203 
(303) 339*2232 



CONNECTICUT: 



Recently passed Public Act 128 lists required subjects 
to be taught in the public schools* Speaking is among 
those subjects* There is an indefinite plan to assess 
speaking or listening on a statewide basis in the 
future* 



DELAWARE: 



Contact s 

George D* Kinkade, Consultant 

Bureau of Research, Planning and Evaluation 

State Department of Education 

Box 2219 

Hartford, CN OS 115 
(203) 566-7232 

Has a .well developed state program titled 'Delaware 
Educational Assessment Programs* initiated in 1972 and 
currently requiring the annual administration of 
nationally standardized, norm-referenced achievement 
tests in reading, language arts, and mathematics* 
Currently the program uses the California Achievement 
Test* Speaking and listening are mentioned as 
"goals", but no plans are currently being made to test 
these areas* 



FLORIDA 
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Contact: 

Robert A* Bigelow 

State Supervisor of Educational Assessment 
State Department of Public Instruction 
Townsend Building, Box 1402 
Dover? DE 19901 
(302) 678-4583 

The state has a well developed program for testing 
certain basic skills as a requirement for graduation, 
but speaking and listening are not among them* The « 
state, basic skills plan mentions oral communication, 
but does not specify objectives, curriculum materials, 
or assessment procedures* Speaking and listening 
currently do not have a high priority* 

Contact : ^ 
Ms* Ada Puryea; 

Ad^nistrator of Early Childhood & Elementary Education 
Florida Department of Education 

Inott Building : 

Tallahassee, FL 32301 



First, the State Board of Education policy statement 
30-770 and 30-710, •High School Graduation 
Requirements, (Nov. 1976 and amended in 1978 and 
1980) , which goes into effect with the graduating 
class of 1984, requires that each student demonstrate 
at least minimal mastery of the following competency 
performance standards: reading, writing, mathematics, 
speaking and listening, and problem solving. For 
speaking and listening, the requirements readi A 
student will demonstrate competence by his or her 
ability to receive and transmit oral and aural 
communication in the context of academic problems, 
everyday tasks and employment activities. Indicators 
of speaking and listening may include interpreting 
aural communications, composing oral directions and 
questions and using formal and informal speaking 
styles. 

Local authorities have responsibility for providing 
evaluation services together with appropriate 
curriculum. < . 

Second, the State Department of Education is preparing 
a document called "Comprehensive Language Arts Guide" 
for distribution to schools*? This Guide should be 
available in June 1981. 

Contac t: 

Dr. William Hammond, Director 
Georgia Basic Skills Program . 
Georgir Department of Education 
156 trinity Avenue 
Atlanta, GA 30303 
(404) 656-2584 

(A bright spot.) Board of Education Regulation 
#4540.1, revised 8/78, effective with graduating class 
of 1983. They have developed specific performance 
expectations for grades 3, 6, 8, 10 and 12 in language 
arts, including. many related to speaking, listening, 
and nonverbal communication. A test titled "Oral 
Communication Test for Essential Competency #4"" has 
been drafted and field tested, in April/May 1980. The 
tfest currently contains a rating scale similar to the 
one developed for Massachusetts. Plans call for 
further development through the 1980/81 school year. 

Contacts 

Dr. Selvin A. Chin-Chance 

Administrator, Test Development 6 Administration 

Section 
Department of Education 
P. 0. Box 2360 
Honolulu, HI 96814 




Proficiency testing is being dons in the Idaho public 
schools on a district optional basis* Approved 
2/3/77 9 it defines basic skills as reading, writing, 
arithmetic, and spelling c Nothing is included on 
speaking and listening and there are no current plans* 

Contact: 

Karen Underwood, Consultant 

Proficiency Testing 

State of Idaho 

Department of Education 

Len B* Jordan Office Building 

Boise, ID 83720 

(208) 384-2112 

A planning document, prepared by, the Illinois Speech 
and Theaterjkssociation'in cooperation with the 
Illinois State Board of Education Department of 
Program Planning and Development, outlines four major 
categories of oral communication skills and describes 
specific communication skills for each* No assessment 
method is suggested* . The four major categories are: 
1) providing a wide variety of communication 
situations; 2) learning to analyse communication 
messages a ppropriately; 3) developing appropr iate 
communication strategies; and 4) developing the 
ability to evaluate messages. 

A copy of this plan is in the committee files* 

Contact: 
Mine Halliday 

SHOE Program Planning- and Development 
State Office of Education 
100 North First Street 
Springfield, IL 62777 

No current statewide efforts until 1983 to either 
identify speaking and listening skills or to 'specify 
assessment procedures* Some local districts have . 
excellent programs in speaking and listening/ most 
notably the Gary Community School Corporation* 
Commencing with .the graduating class of 1980, all 
students must pass an oral proficiency examination* 
All middle school students are required to take a 
minimum of one semester of speech* All high school 
sophomores are required to take a full. semester of 
speech* An evaluation instrument accompanies the 
courses: It rates eight areas holistically on a four 
point scale* The eight skill areas are: 
articulation, pronunciation,* verbal utterances, rate, 
word usage, voice qualities, volume, and sentence 
structure* 
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INDIANA: (continued) • 

Contact : 

Dr. Patrick Gavigan 

State Department of Public Education 

120 West Market Street 

Indianapolis, IN 46204 

(317) 633-4355 



IOWA: 



Currently ho testing of speaking and listening and no 
priority to do so. State Department of Public 
Instruction does have a state plan to train teachers 
in the assessment of aath, reading, and writing, as 
well as oral communication. 



Contact : 
Lee Wolf 

State Department of Public Instruction 
Grimes State Office Building 
Des Moines, IA 50319 
(515) 281-3190 



KANSAS: 



Nothing at this time in speaking and listening, 
being developed for reading and mathematics. 



Tests 



KENTUCKY: 



Contact:, 
Mr. Mel Riggs 

Education Program Specialist 
State Department of Education 
120 East 10th Street 
Topeka, KS 66612 

r 

Nothing currently planned on assessing speaking and 
listening. Has a criterion- referenced test for 
reading, writing, and arithmetic. 



Contact : 

Dr. Charles B. Morgan 
Director, Testing 
State Department of Education 
Capitol plaza Tower 
Frankfort, KY 40601 

LOUISIANA: In 1977 Louisiana established a program of minimum 

standards in reading, and later (1979) in writing and 
mathematics. Speaking and listening are not treated 
separately; however, each previously mentioned skill 
area includes some oral communication skills. Por 
example, reading includes, for kindergarten and first 
grade, skills in auditory discrimination, oral 
vocabulary, ortal expression of ideas, and listening 
comprehension • The higher the grade, .however, the 
less attention is given to oral communication. 
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LOUISIANA: (continued) * 



Contact: 

Nr. JoMph Williams 

Director , Planning and Evaluation 

Bureau of Assessment 

State department of Education , o 

P. O. Box 44064 
Baton Rouge r LA 70804 

MAINE: By 1981, the State Department will have a plan 

suggesting models districts can use in developing 
their own plan. By 1982, each school district must 
have a written plan for the achievement of basic 
competencies. Currently the State Department of 
Education is developing a list of competencies in all 
areas, including speaking and listening. However, it 
seems unlikely there will be performance testing, most 
probably the assessment will be paper and pencil. 

Contact : 
Dale Douglas 
Educational Planner 
State of Maine 

Department of Educational and Cultural Services 
Augusta, MA 04333 

MARYLAND: ; No plans to test speaking and. listening. Currently 

only testing reading. 

Contact : 

Dr. William Grant, Chief 
Program Assessment Branch 
State Department of (Education 
BWI Airport 
P. 0. Box 8717 
Baltimore, MD 21240 
(301) 796-8300 

MASSACHUSETTS: Has a well developed plan for assessing speaking and 

listening. The speaking test is currently undergoing 
refinement. Plans are to have it in place for the 
eighth grade level by the' school year 1981-82; the 
listening test should be finished by the 1982-83 
school year. 

Contact :, 
Tracy Libros 

Bureau of Research and Assessment 
State Department of Education 

31-St. - James 

Boston, MA 02116 
(617K 727-0190 
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MICHIGAN: 



PL#38 called for statewide assessment in language 
arts, reading, etc. Zn 1973, criterion-referenced 
assessment was mandated. She Department of Education 
developed objectives for the state in 1975-76, 
including objectives for speaking and listening. In 
1977, the state began to assess listening (speaking 
not yet done, though ready to go) . Listening test 
used on 4th, 7th grades in 1978f on 10th •in 1979— ~ 
Used statewide sample (1,000 - 3,000 per grade 
level) . Test is mandated but is not required for 
graduation. Bight listening skills (each criterion- 
referenced) and twelve speaking skills (not yet field 
tested)/ are used. Deals with using videotape, 
pantottiming/roleplaylng, public speaking, and small 
group discussion. State has instructional packages on 
bow to teach and assess these areas. 



MINNESOTA: 



Contact : 

Ed Roeber 

Supervisor/MEAP 

State Department of Education 

620 Michigan National Tower 

P. 0. Box 30008 

Lansing, MI 48909 

(517) 373-8393 

Minnesota passed a state basic skills law in 1979 that 
requires school districts to develop programs in five 
basic skill areas: writing, listening, speaking, 
reading, and computing. "Essential, learning outcomes 0 
for each of the five basic skill areas have been . 
developed. The outcomes for listening and speaking 
are currently undergoing revision and are expected to 
be finalized in May, 1981. > The outcomes are used for . 
instructional purposes, only: there are no statewide 
tests in any of the areas used for graduation 
requirements . 



MISSISSIPPI: 



Contact : 

Ms. Patricia Koran 

Basic Skills Supervisor 

Minnesota State Department of Education 0 

550 Cedar 

St. Paul} Minnesota 55101 
(612) 297-2657 

3o plans for assessing speaking and listening. 



Contacts 

Mr. Rex Pouncey 

Supervisor, Accountability and Assessment Programs 
State Department of Education ~ 
P. 0. Box 771 - 
Jackson, MS 39205 



ERJC 



5616D 



16 



21 



In 1979, the State Department of Elementary and 
Secondary Education developed tx "Guide for the Basic 
Essential Skills Test." The Guide describes three 
basic skills; reeling/language arts, mathematics, and 
governeent/economics. Speaking and listening are not • 
directly addressed, though there are four specific 
objectives under Re&ding/Language arts that relate to 
oral communication. These are; . (number is objective 
number in Guide) 14) apeak and write effectively in 
different social and business situations and with 
persons of varied ages, or backgrounds, 17) recognise 
fchtt main idea and specific details in an oral 
presentation, 19) communicate with persons of 
different age groups and in different situations by 
reading -orally $ and 21) follow oral directions to • 
complete a process* The g^ide "details sample criteria 
for these objectives, but the specific criteria used 
to judge the sk|U is the responsibility of the local 
school district. 

Contact; 

Dr. John J. Jones 
Supervisor of Assessment 

State Department of Elementary and Secondary Education 
P. 0. Box 48Q 

Jefferson City, Missouri 65102 

Nothing currently in speaking and listening. 

Contact ; 

Claudette Johnson, Consultant 
English/Language Arts 
Office of Public Instruction 
State Capitol 
Helena, NT 59601 

»> 

Currently state regulations call for assessment in 
reading, writing, and computation, but nothing. in 
speaking and listening. 

Contact ; 

Esther Arganbright 

State Department of Education 

Box 94^87, 301 Centennial Mall South 

Lincoln, MB 68509 

(402) 471-2295 

Current plans call for selection of instrument by 
5/V81 and yeptly assessment* beginning 10/81. .Oral 
communication skills are not yet defined. 




- NEVADA: (continued) 



ton 
Her] 



itacti. 



Serbert R. Staff ens, Director 
Planning and Evaluation 
ftevada Department of Education 
C^oitol Complex 
Carson City, HV 89710, 
(702) 885-5700 



MEN HAMPSHIRE i Hew Hampshire completed an assessment of listening 

skills in October. Students in 5th, 9th, and 12th 
^grades—about 600-700 students at each level—Were 
assessed (multiple cassette messages, around 45 
questions) • Mo plans to do a random sampling 
assessment on speaking, but Dr. Baker (below) is 
working on 2-3 individual school district projects. 

- Contact : 

Dr. Joanne Baker 

Consultant, English Language Arts 
State Department of Education 
Division of Instruction 
64 N. Main 
Concord, NH 033Q1 
(603) 271-3747 



. MEW JERSEY: 



Has plans to develop a speaking and listening 
assessment instr ment sometime after this school 
year. No definite timetable as yet. 



Contact i 

Lydi^a Greenberg, Coordinator 
State Testing Program 
Department- of Education 
225 West State Street 
Trenton, MJ 08625 
(609) 792-5181 



NEW MEXICO: 



t N 4 



Has a -well developed plan for basic skills • However, 
basic skills are defined in the plan as reading, 
writings arithmetic, and problem solving. No current 
plans to include speaking and listening. 



Contact; • 

Howard J. Scheiber 

Director of Writing Assessment and Pedagogy 
Office of Basic Skills 
Department of Education 
Santa Pe, NM ' 87503 
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Students are tested for final competency in grades 11 
or 12. Final competency via the state Board of 
Regents exam is required for high school diploma. 
Speaking and listening are* not identified as part of 
the exams which currently focus on .reading, writing, 
and mathematics. 



Dr. Windsor Lott, Chief 
Bureau of Elementary and Secondary Educational Testing 
State Department of Education 
Washington Avenue, Room 771 
Albany, NT 12234 
(518) 474-5099 

No plans to assess speaking and listening. 
Contact : 

Dr. William, J. Brown 

Director of Research 

State Department of Public Instruction 

Raleigh, NC 27611 

(919) 733-3809 

The state basic skills plan is still ir process. The 
current draft does not specifically address speaking 
and listening, but focuses most of its content on 
reading and mathematics. 

Contact : 
Ethel Lcwry 

State Department of Public Instruction 
State Capitol Building 
Bismark, ND 58505 
(701) 224-2292 

No statewide assessment or minimal competencies 
mandated. No one looking into speaking or listening. 

Contact : 
Jim Pay ton 

Education Concultant, Planning and Evaluation 
State Department of Education 
65 South Front Street, Room 804 
Columbus, OH 43215 - 
(614) 466-4838 

No current plans to assess speaking and listening. 

Contact : 

Tommy Hodges, * 

Administration, Office of Accreditation 
State Department of .Education 
2500 North Lincoln, Room 245 
Oklahoma City, OK 73105 
(405) 521-3333 9 





\ 



OREGON* The Oregon State Board of Education adopted six 

instcuctionally-related state goals* The first goal 
states "Each individual Will develop the basic skills 
of reading, writing, computation, spelling, listening, 
• ^ speaking, and problem solving. " The Oregon Basic 
flkills Plan states it has very little data on oral 
csanunicationV All school districts in Oregon are ' 
required tb have written district, program and course 
\ goals in the areas of reading, writing, speaking, and 

computing. The local districts have quite a bit of 
flexibility in developing these goals. 



5 



PENNSYLVANIA: 



Contact: . 
Dr. Gordon Asher * f 
State* Department of Education* 
700* Pringles Parkway, S. B. 
Salem, OR 97310 

One of the bright spots. Tt\e -state has published a 
working copy of "Speech in the Classroom" which 
includes a well devalued set of teacher rating scales 
for assessment of speaking skills. The* assessment 
program is instructional and is designed to help the 
classroom teacher identify areas for instructional 
development in speaking skills, Ix) addition, a 
companion document titled "Communication in the 
Classroom 19 suggests theoretical and curriculum 
materials for the classroom teacher. Both the rating 
scales and the curriculum materials are formed for 
different grade, levels,. The assessment procedures are 
primarily descriptive of types of communication 
instruction that might take place in the classroom* 
Ho plans to use this material for anything but 
instructional purposes* 



Contact : 

Evelyn w. Miller 

Bureau of Curriculum Services 

Pennsylvania State Department of Education 

333 Market Street 

Harriaburg, PA 17126 

(717) 783-3947 * 



RHODE ISLAND: 



There are no plans for statewide assessment at the 
elementary level. * At the secondary levels the area of 
language arts includes objectives for speaking and 
listening* A test-items bank includes sets of items 
(in the "life skills" area) that supposedly assess 
speaking/listening* Since these are very expensive to 
administer and score, there* are no plans to use these . 
items for a couple of years at least. 
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Contacts , 
Martha Highamith 
Statewide Assessment Program 
Department of Education 
199 Promenade Straat 
Providence, RZ 02908 
(401). 277-3126 

Stata has bagun a process to implement an annual 
atatetrida assessment program for K-12. Kindergarten 
and primary grades currently hav% more oral akill 
items (both apaaking and listening) than do later 
grades which focus more on reading and writing. Many 
of the items for the lower grades require an> oral 
response. Elementary grades* have been field tested; 
first full administration in Spring 1981. Secondary 
levels will be field tested in Spring 1981. 

Contact! _ 

Or. Vana H. Meredith 
Chief Supervisor - 
Educational Assessment Section 
State Department of Educate on 
1429 Senate Street, Room. 604 

Columbia, SC 29201 _ i • 

South Dakota has developed thai'* Language Arts 
Curriculum Guide: K-12,* ^published in 1978. This 

/guide* is to b* used by local 'districts and individual 
teachers to assist "theto in assessing and planning 

.local' language arts curriculum. The guide is. divided 
into the following grade levels: ' Primary, K-3; 
Intermediate, 4*6; Junior Higfi, -7-9j an£ Senior Highi* 

' 10-12. Concerning listening and speaking/* the guide 
states: "listening and f speaking skills have been 
'identified and emphasized for all the gradeHevels 

.because ^hey are. the basis for developing the skills 
of reading and writing." Samples of .Program 
.Objectives for glistening include: • develops acceptable 

.manners' in listening (primary), recognize^ 
similarities and differences. in phonemes (primary) , 
learns to listen for purposes of recall 
(intermediate) , learns to valued listening as an' 

.integral part 'of the communication process (junior 
high) , etc. Examples of Program Objectives in 
speaking include: . learns to participate without 
anxiety in a variety of verbal situations 
(intermediate), learns to use appropriate and varied 
vocabulary in oral presentations (intermediate) , and 
develops* fluency and poise in formal and informal 
speaking situations. 
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Contact: 

lathy Hughta-Iaan 
> Language Arts Program Director 
Division of Elementary and Secondary Education 
Jtoeif Building 
Pierre, 3D S7S01 
(6P5) 773-3508 

Ho plans to assess speaking and listening* 
Contact: 

Dr. Tern Innesr Director 

Educational Assessment 

State Testing and Evaluation Center 

University of Tennessee 

1000 White Avenue 

Xnoxville, TO 37916 

(615) 974-5385 

Senate Bill 350 amended Sect. 16.176 of the Texas 
Education Code established a criterion referenced 
assessment program for reading, writing and math. It 
does, not address speaking and listening. 

Contact : 

Mr. David Splitek 

Division of Planning and Educational Assessment 
Texas Education Agency 
201 E. 11th Street 
Austin, TX 787*12 
(512) 475-2066 

One of the few states that has a program of testing 
basic skills in place and operating begun in 1977 and 
adopted statewide, the system has a four by six matrix 
of skills. Four "life competencies" including 

. consumer of goods and services, career, health and 
safety, -aiad democratic governance. These four are 
applied to each of six basic skills including reading, 
writing, speaking, listening, computation, and problem 
•solving. The student must demonstrate a competency in 
each of the 24 functioned, competencies defined in the 
matrix**. A handbook provides material on curriculum 
locations, performance indicators, testing procedures 

- "and" instructions^ and required forms. 



4 Contact : v 
Per r in Van Wagoner, Coordinator 
High School 'Programs 
Utah State Board of Education / % K 

250 East rif th South ' . If ' € >* 

* » Salt Lake City, OT 84111 
(801). 533-$888 • 

. • A. - 



Another state with a program In place. ^ The program 
began with the school year 1977-78 . During this year 
95,486 students wars assess ad to determine the level 
of mastery in the basic skills of reading , writing, 
listening, speaking, computing, and reasoning. The 
identified competencies will be required of graduates 
beginning with the class of 1981. The data and the 
discussion of results is contained in a series of 
booklets published by the Department of Education 
titled Basic Competency Program . Speaking and 
listening skills are evaluated by asking the student 
to orally respond to a variety of situations and then 
rated by their teacher. 

Contact t « , 
Shirley Beid^ 

Director of Assessment/Testing 
State Department of' Education 
Montpelier, VT 05602 
(802) 828*3135 

They are developing a- statewide program called 
"Standards fp^ Learning." This program includes 
learning objective's in all the basic skill areas, 
including speaking and listening. A tentative set of 
objectives were sent' out for comment 12/80. Revised 
objectives should be published by simmer of 1981. 
Programs will begin on a trial basis in the Pall, 
1981. Assessment will probably take place beginning 
Fall 1982. 

Contact : 

Dr. Kenneth M».gill, Supervisor 
Jr. High/Middle Schools .» 
Division of -Humanities & Secondary Administration - * 
Department of Education 
P. O. Box 60 - 
Richmond, VA 23216 
(804) 225 r 2053r 

No current plans to test speaking and listening on a 
statewide basis. 'Basic skill program includes oral 
communication, but competencies are not defined. 

4 1 
• *» 

Contact : 

James Click, Coordinator 
Basic Skills Program 

Superintendent of Public Instruction * 
Old Capitol Building 
Olympia, MA 90504 
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WEST VIRGINIA: 



No plans to assess speaking and listening 



Contact: 

Mr. Charles D. Duffy, Coordinator 

Assessment and Testing 

State Department of Education 

Bst. Room B-057 

1900 Washington Street 

Charleston, WT 25305 

(304) 358-3230 ' 

WISCONSIN: Wisconsin presently has a statewide assessment 

procedure for reading, writing, and mathematics. They 
are in the initial stages of developing a statewide 
assessment procedure in speaking and listening. The 
goals and specific skills to be assessed should be 
identified by 8/81, and a pilot test will take place 
approximately 1/82. 

Contact : 

Mr. Ben Brewer 

Coordinator Basic Skills 

Department of Public Instruction 

125 South Webster Street 

Madison, WI 53702 

(608) 267-9268 

WYOMING: Currently Wyoming has no plans for any statewide * 

testing program in any area. Their Basic Skills 
Program, written in response to Sections 222 and 224 
of Title II BSSA includes speaking, and listening but 
the plan does not. specify competencies or evaluation. 

Contact : 

Alan Wheeler, Director 
General Program Unit 
Department of Education 
Hathaway Building 
Cheyenne, WY 82002 
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ASSESSMENT OP COLLEGE LEVEL SPEAKING AND LISTENING SKILLS 

j 

Rebecca B. Rubin 
University of Wisconsin-Parkoide 



ABSTRACT 

The aim of this paper is to describe the present conditions affecting 
assessment of speaking and listening skills at the college level. The 
paper covers the "state of. the art" in college level programs , 
concentrating on the development of the Communication Competency 
Assessment Instrument, This instrument has undergone extensive 
reliability testing at the University of Wisconsin-Par kside. It is 
predicated on the Speech Communication Association's "Guidelines for 
Minimal Competencies in Speaking and Listening," which identifies four 
competency areas: communication codes , oral message evaluation, basic 
speech communication skills, and human relations. It also follows the 
SCA' s "Criteria for Evaluating Instruments and Procedures for Assessing 
Speaking and Listening Skills," making it one of feor instruments that 
directly assess college level speaking and listening skills. Initial 
coefficient alphas upport the reliability of the instrument; inter-rater 
reliability coefficients uphold the use of the rating booklet, which 
provides five skill levels for each of the 57 assessments (three 
assessments for each of 19 specific competencies). Short forms of the 
test are in their initial stages; judgments about the viability of using 
short forms are presented. Inc?additiqn, conclusions about the need for 
college level assessment instruments are offered. 
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ASSESSMENT OT COLLK3R LEVEL SPEAKING AND LISTENING SKILLS 



In March 1978, AgMiilw Functional Communicati on (Larson, Backlund, 
Bateond t Barbour, 1978) was published; this text detailed tba stata of 
the art in com unlcafcion competency tasting to datt.u Tha authors -defined 
a number of main content areas, such as Developmental Language and 
Communication Skills, Communication Competence and Appropr i&tenesa , 
Receiving— Listening, Anxiety-Apprehension, and Interaction 
Descriptions. i. total of 53 of tha 90 tests reviewed vara appropriate 
foe collage level or adult parsons. However, no one test provided a 
comprehensive evaluation of speaking and listening skills) many of the 
instruments were specialised assessments of competencies such as anxiety,, 
counseling/ interviewing techniques, personal maturity, social competence, 
and group interaction* 

At that time, the University of Wisoonsin-Parkside had given up its 
search for an instrument to assess speaking skills, an area originally 
included in a comprehensive Collegiate Skills Program assessing, writing, 
mathematics, reading, research paper development and library skills — all 
skills which students should have mastered by the time they have earned 
45 college credits (Maeroff , 1978) . The lack of an existing 
comprehensive assessment instrument in communication and concern over 
minority bias seemed to be the major reasons, for "tabling" the speaking 
covenant* With a grant from the -University of Wisconsin system (Rubin, 
1980s) , this author aet out to discover the state of the art in college 
level competency testing, a prelude to developing a communication test 
for the UW-Parkside program* Pew assessment instruments at the college 
level were found. 

The American College Testing Program's College Outcome Measures 
Project (CQMP) battery provided an assessment of general education 
knowledge and skills; one of the six areas assessed was labelled 
"Communicating," the "ability to send and receive information in a 
variety of modes (written, graphic, pral, numeric and symbolic) # within a 
variety of settings (one-to-one, in small and large groups),' and for a 
Variety of purposes (for example, to inform, to understand, to persuade, 
and to analyze)" (Steele, 1979, p. 2). The purpose of the test was to 
assess effective functioning in adult roles in society. The "Speaking" 
port ion. of the Communicating area. tended to concentrate on delivery and 
discourse; the test did include direct measures of speaking ability* 
However, the Collegiate Skills Program at the University of 
Wisoonsin-Parkside was more concerned with the student's ability to 
function in college contexts, rather than in society at large. 

Also examined was the program at Alverno College, which is more 
aabitiou* than most colleges can afford (Alverno College Faculty, 1976) . 
At Alverno, communication ability is defined as the ability to write, 
read, speak, listen, use media, and use quantified data. The speaking 
portion assesses a student's ability to speak extemporaneously, clarify 
the setting and context, use effective delivery and linguistic ~ 
conventions, organize and develop ideas, use visual aids, and evaluate 
bis or her own strengths and weaknesses. There are 6 .levels of student 
proficiency which can be attained (LnJ a combination of individualised and 
group testing sessions and coursesw ^gainj a„ ve?y ambitious program, but 
a broad mix of skill areas, some ot which are, not speaking and listening 
per se« / / _ ' 





Tbt Speech Communication Association's 1978 Task Fores on Minimal 
Speaking/Listening Competencies presented the Minimal Competencies in 
Speaking and Listening for High School Graduates (Bassett, tfhittington, & 
Staton-Spicer,. 1978) • This document seemed highly appropriate and 
relevant to this task of creating a oollege-level assessment instrument. 
It identifies four main competence areas: Communication Codes (ability 
to use and understand spoken English and nonverbal signs) , Oral Message 
Evaluation (ability to use appraisal standards to judge oral messages and 
their effects), Basic Speech Communication Skills (ability to select and 
arrange message elements to .produce spoken messages) , and Human Relations 
(ability to maintain interpersonal relationships) • These four main 
competence areas are further divided into 19 specific competencies, with 
examples. of application within three contexts: occupational, 
citizenship, and maintenance* For purposes of the Communication . 
Competency Assessment Instrument (CCAI), a fourth context — educational — 
was created, along with three application examples for each of the 19 
competencies (see attached Table 1) . These application examples were 
used as the basis for the first version of the CCAI* There were a total 
of 57 possible assessments to be made about a student's ability to 
function in specific educational environments; in class, and with 
instructors, fellow students, and academic advisors. (Table 1 - see 
appendix) 

Meanwhile, the Speech Communication Association endorsed and 
published the Educational Policies Board's "Criteria for Evaluating 
Instruments and Procedures for Assessing Speaking and Listening* 
(Backlund, Brown, Gurry, a Jandt, 1979) ; these criteria were followed in 
developing the CCAI* The Board document asserts that assessment 
instruments should be "Valid, reliable, and feasible. Specifically, it 
proposes the following guidelines: stimulus materials, should require the 
demonstration of a skill; inferences about a speaking or listening skill 
should not be made from tests requiring reading and writing; the % 
instrument should be unbiased; the test should assess skills occurring in 
familiar situations and in a variety of communication settings; tests 
should permit a range of acceptable responses; instruments should be 
standardised so that the test administrator's skills will not affect the 
results; the stress level should be equal to that of the setting in 
question; procedures should be practical in terms of cost and time and 
should involve simple equipment; and assessment should be suitable for 
the individual's developmental level* 

. At about the time of this publication, researchers from the 
University of Oklahoma were developing a communication assessment 
instrument to determine a student's admission into .their business program 
(Scafe a Siler, 1979)* They, too, followed the Speech Communication 
Association's lead on the competency areas, but opted for indirect 
assessment: that is, a written objective test. Direct assessment 
procedures were chosen as most appropriate for the CCAI. 



Methods 

The result was a 57-item version of the CCAI (Rubin, 1980b) • There 
are three main sections to the test* The first asks the student to 
present a three-minute extemporaneous persuasive talk on a topic of 



56i60 . s ^32 

-28 ' 



interest, during which the student* a speaking ability is judged on many 
factors (e.g. , volume, rate, clarity, gestures) . next, the student views 
a videotaped class lecture, and is then immedictely questioned about the 
. lecture and asked to respond in various ways to statements about his/her 
experiences in an educational environment. All student responses are 
either oral or nonverbal in nature} writing and reading abilities .are not 
asse s sed. The test assesses only the student* s ability to communicate 
through "Speech and nonverbal actions and to listen. At this stage of 
development, the OCAZ took approximately one hour per student to 
administer. 

This version of the test was critically examined by a number of , 
persons and subjected to numerous, refinements (Rubin, 1981) . Content , 
validity was achieved by presenting five communication faculty members 
with the 19 competencies and 57 assessment items arranged in random 
order; they were asked to place each assessment item into one of the 19 
competency categories. Five questions that initially failed to meet the 
established 80 percent agreement standard were subsequently rewritten and 
correctly categorised by all evaluators. The questions were further 
refined, and a rating book with five leve'a of proficiency for each of 
the assessments was constructed, evaluated, and revised. Four faculty 
members trained as judges used the rating book to evaluate three students 
who had been videotaped earlier while taking the OCAI. A mean-interrater 
correlation of .83 was attained, attesting to the reliability of the 
rating book. The communication skills of 77 students were assessed with- 
this test version, each test session requiring about one hour. 

As one might imagine, an hour's testing time per student presents 
monumental difficulties with large-enrollment institutions. Thus, the 
feasibility of reducing test size was investigated. Coefficient alpha 
analysis was performed on the 57- item OCAI. An overall alpha of .83 was 
achieved. To create a test of approximately one-third the number of 
assessment items (and one-half the administration time) , the least 
consistent items were eliminated. That is, for each of the 19 competency 
areas, the sect valid item was determined and the coefficient alpha for 
this 19-item short form was .79. It is extremely difficult to achieve «, 
.80 coefficient alpha with leas than 30 test items (Nunnaliy, 1978); 
thus, a .79 is considered respectable. However, to increase the 
coefficient alpha, four additional items w»/re added to the 19-item short 
form, the four next highest ranking items, one in each of the four main 
sections of skills. The coefficient alpha for the 23-item form was .82. 
Additional testing was then necessary. 

lighty-three students who were enrolled in the basic communication 
course at the University of Wisconsin-Par kside completed the 23-item 
short form during February 1981 (see footnote 1) . A shortened version of 
the listening videotape stimulus was prepared. It had the same content 
of the original, minus unnecessary text (see footnote 2) . All test 
administrations lasted 30 minutes or less. The data on these 83 students 
were combined, with the data on the first group (77 students) , and again 
submitted to coefficient alpha analysis. The coefficient alpha for the 
23-item test was .80; for the 19-item test, the coefficient alpha was 
.78. It was determined that the two short forms were sufficiently 
similar not to warrant the inclusion of the four additional items which, 
in actuality, had no basis for inclusion. The following results pertain 
to the 19-item version of the OCAI. 
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Remits 



Table 2 shows frequency distribution of total scores on the 19-item 
test short font. The theoretical •passing* grade o£ 3.0 was used as a 
cutoff, the program proposed at UW-Parkside would declare all those 
receiving 57 total points (average grade of 3 for 19 items) as 
•competent"; those above 48 points (2.5 on a 5-point scale) as "in need 
of re m edial self-paced work*; and those below 48 points "in need of 
formal training in a remedial course.* The mean, median, and mode fell 
at about the same point, indicating' a somewhat normal curve for the 160 
students tested (see attached Table 2). 

Initial analyses suggest that the instrument is free from bias. Hale 
and female students did not appear to differ in their abilities to pass 
the test on the first trial (X 2 - 0.68, df - 1, n.s.). Similarly, no 
differences were found between minorities (hispanic, black) and 
non-minorities (X 2 ■ 3.33, df » 1, n.s.). However, there was a 
significant difference between minority (X - 56.13) and non-minority (X « 
63.74) students (t - 3.53, df - 158, p< .001, two-tailed) on their total 
scores. At first glance, it appears as though the few minority students 
who did poorly on the test affeqted the mean o^the minority student 
group (H ■ 16, s.d. ■ 9.75), moreso than non-minority students who did 
not excel (H - 144, s.d. - 8.00). Additionally, the non-random sample of 
students reflects students who sometimes select the basic communication 
courses to improve known inadequacies in communication skills. This may 
have bad an influence on. the means. 

Other data collected on the students, summarised in Table 3, included 
academic major, number of college credits completed, age, grade point - 
average, number of communication or speech courses taken, and pact 
speaking experience. Briefly, differences were found on the 19-item test 
fort (a) business and humanities majors; (b) those with 30 or fewer 
credits and those with over 60 credits; (c) the under-20 age group and 
the 20-24 age group; (d) those with a low GPA and those with a high GPA; 
and (e) those with little or no speaking experience and those with quite 
a bit of experience. Correlations of the raw scores of these data are 
found in Table 4. These analyses suggest that a variety of experiences 
and abilities leads to communication competence, as defined by the OCAI. 
Indepth investigation of these data is in progress (see attached Tables 3 
and 4). 

Additionally, each of the 19 items, of the CCAI short form was closely 
examined. As Table 5 shows, 11 percent of the students tested had 
problems asking a question; 33 percent .could not organise ideas well; 32 
percent could not give accurate directions; 35 percent could not 
adequately express and defend a point of view; 10 percent didn't 
understand the difference between a fact and an opinion; 27 percent could 
not understand suggestions for improvement presented by an instructor; 14 
percent could not adequately identify the work to be performed on an 
assignment when it was presented orally in class; and 49 percent could 
not describe the point of view of a person who disagreed with them (see 

.. attached Table 5) • 
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Discussion 



This Initial research effort has disclosed that for some students $ . 
oacmunication problems oould vary nail inhibit learning. It is 
iterative that measures be .developed to help students identify these 
problem and improve their communication skills so that they will not be 
disadvantaged learners* The State of Florida (Walker* 1979) is at work 
on this issuer as is Phil Backlund at Central Washington University. 
Valid and reliable procedures are needed to help students determine and 
correct their speaking and listening deficiencies. 

Classrooms are communication arenas where students and teachers 
interact by, communicating, Ability to survive "in these environments is 
essential to successful college completion. The CCAl is proposed as a 
method of assessing the basic survival skills students need. The results 
cf college-level competency testing can be used ir a variety of ways. 

It may be beneficial to some colleges to establish individualized 
instruction programs to help students improve specific skill areas. 
Other colleges may be able to identify specific courses which would help 
students improve their skills. And still other institutions could use 
the results in advising students about their weaknesses and helping them 
plan future course-work. Whatever the result* college-level competency 
assessment can provide useful information and feedback on students* skill 
achievement to help them become more effective, communicators in their 
college classes. 



31 



°5 



POOTOOTES 

« v 

*f his course is required only for Ccmunlcation majors (Humanities) ; all 
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Table 1 
•Educational Cohtext" 



COMMUNICATION CODES 

A* LISTEN EFFECTIVELY 10 SPOKEN ENGLISH. 

1. Understand directions given by c classroom instructor for 
class assignments . 

2* Understand mattrial presented in a class lecture* 
3. Understand an instructor's suggestions for improving 
one's abilities* 

B. USB WORDS, PRONUNCIATION AND GRAMMAR APPROPRIATE FOR THE 
SITUATION* 

!• Use appropriate language in a classroom report. 

2. Use appropriate grammar when speaking to others. 

3. Use pronunciation which is understood by others* 

C. USE NONVERBAL SIGNS APPROPRIATE FOR THE SITUATION* 

1* Use appropriate gestures and eye contact when interacting 
with others* .. 

2. Use appropriate facial expressions and tone of voice when 
conversing with one's instructor or fellow students. 

3. Recognize and/or use appropriate gestures, eye contact, 
and facial expressions when communicating understanding 
or lack of understanding in a listening situation. 

c 

D. USB VOICE EFFECTIVELY* 

!• Css appropriate rate when making a report in class* 

2* Speak loudly enough to be heard in a classroom situation* 

3. Use appropriate clarity when speaking with others. 

ORAL MESSAGE EVALUATION 

A* IDENTIFY MAIN IDEAS IN MESSAGES* _ 

*? 

1* Identify the work to be- performed when the assignment is 

given' orally in class* 
2. Recognize performance standards for work assigned orally 

in class* 

3* Identify the main ideas in a class lecture* 



DISTINGUISH FACTS FROM OPINIONS. 

1* Recognise an opinion in a class lecture or report. 
2. Recognise a fact in a class lactura or report* 
3* Distinguish between facts and opinions in an 
intarparaonal interaction* . . 

DISTWGOISH BETWEEN INFORMATIVE AMD PERSUASIVE MESSAGES. 

1* Distinguish between informative and persuasive massages 

in a class report* 
2. Distinguish between inforaativa and persuasive aassagas 

in a class lactura* 
3* Distinguish batman inforaativa and parsuasiva messages 

in an intarparacnal intaraction. 

♦RECOGNIZE WHEN ANOTHER DOBS MOT UNDERSTAND TOOK MESSAGE* 

1* Racognisa whan an instructor or fallow classmate doesn't 
understand the question you are asking* 
Recognise when an instructor or fellow classmate doesn't 
understand the question you are answering* 
3* Recognise whan others do not understand your explanation 
of a concept reported on in class. 

" - - — 

C SPBBCE COMMUNICATION SKILLS 

i 

EXPRESS IDEAS CLEARLY AND CONCISELY* 

o 

lm Make a clear report on a subject of interest or one 

you 9 ve researched* 
2. Concisely explain course requirement^ to a new student in 

class* 

3* State clearly your reasons for taking a particular course 
EXPRESS AND DEFEND WITH EODENCE TOOK POINT OT VIEW* 

fK 

1* Express and defend ykur view in a classroom report* 

2. Express and^ defend your suggestions for improvements in 
your school. 

3* Express and defend your position th^t a grade you 
received was incorrect* 

ORGANIZE (ORDER) MESSAGES SO THAT OTHERS CAN UNDERSTAND THEM* 

1* Use a chronological order to explain your activities 

throughout the day* 
2* Use a topical order to explain a course you took last 

semester* 

3. Use a problent-cause-solution order when discussing with 
an adviser/counselor an academic problem you are having* 
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D. ASK QUESTIONS TO OBTAIN INFORMATION 

1. Obtain Information about raquiraaenta tot your major." 
- 2. ObUin information about how to ooaplata an eaaignaant. 
3. . Obtain auggaationa about bow to iapr ova. your daaarooa 
parforaanca. • ... * 

• < i 4 * 

B. ANSHBR QQ2STI0NS SFFBCTITSUF t • 
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1*. Answer an instructor's questions about your claslj 

petformance. 
2. Answer a question based on a class lecture. 
3« Answer a question asked, by a classmate About a course you 
• are both taking* • 

Fa GIVK CONCISE AMD ACCURATE DIRECTIONS • 

la Direct fellow students in performing unfamiliar tasks or 

to an unfamiliar location. 
!• Instruct a new student on how to do well in college 

passes* 

3* Give accurate arid concise directions to others* 
G. SUMMARIZE" MESSAGES. ^ 

1. Summarize oral instructions given by' an instructor. 

2. Summarise a class . lecture. 

3. Give a summary of students' suggestions to an instructor. 
HUMAN RELATIONS 

A. DESCRIBE ANOTHER'S VIEWPOINT. 

1. Describe the viewpoint of an instructor who disagrees 
with your evaluation of your classroom performance. 

2. Describe the viewpoint of a fellow student who disagrees 
with your evaluation of a clasfc you've both taken. 

.3c Describe the position taken on an issue by an instructor 
or fellow classmate with which you disagree. 

B. DESCRIBE. DIFFERENCES IN OPINION. « 

* • 

1. Describe differences in opinion with fellow students 
about course related issues » 

2. Describe differences in opinion about the steps necessary 
to accomplish your academic or vocational goals. ' 

3. Describe differences of opinion which occurred in a class 
discussion. 
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KXPBBSB RZLXMGS TO OTHERS* 

* • » 

1. Express satisfaction or dissatisfaction to an instructor 

about a course you have taken. 
2* Express feelings of satisfaction or dissatisfaction about 

working with otbars on group projects for classes* 
3 r * Express eepathy to a friend who has not done well on a 
" class assignment or in a course. 

nnonc soc^/eitoam. ■ . 

1. Introduce yourself at the "beginning- of the semester in 
class. 

2. Request an appointment with a counselor or adviser. 

3. Conclude a conversation with an< instructor. 
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TABLE 2 



N » 7 

« 

4.375% 



RELATIVE FREQUENCY OF 160 STUDENTS ON THE 19-ITEM SHORTFORM 

■ * v 'Standard * > Score. on 

Score * Frequency . Deviation * - N 5-pt. scale 

39 1 ' 

40 . ,1 . r 
44 • 1 

* . 45 -1 

46 1 -2 # 

47 'i- ' A 



2.5 



.48 1 

49 3 

N « 33 SO 1 

... 51 3 

20.625% 52 1 

53 ' j 5 

54 " 5 

55 4 

56 10 



-1' 

- 3.0 



57 3 

58 6 

59 3 

60 3 

61 9 

62 3 _ 
.5 63 8 X 

*** 64 18' - 

65 7 

66 4 3.5 
N « 120 67 8 

68 6 

75.000% 69 1 

70 13 

71 4 +1 • 

72 5 

73 4 

74 4 

75 2 

76 1 4.0 

78 3 • 

79 2 

80 1 +2 

82 1 

83 1 

♦Mean * 62; 98 **Median « 63.78 ***Mode » 64.00 
$.d. - 8.475 range » 44.00 
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ANALYSIS OF VARIANCE TABLES FOR THE 19-ITEM TOTAL SCORE 




• Mean* 


N 




at 


MS £_ 


E 


ACADEMIC MAJOR 














Science 


63.50 


10 


between 


8 






Humanities 


65.95a 


39 


groups 


170.31 2.56 


.012 


Fine Arts \ 


60.80 


5 










Behavioral Sci. 


6S.60 


10 


within 








Business 


S9.7Sa 


44 


groups 


151 


66.61 




Computer/Eng. 


67.60 


5 










Social Science 
Education 


69.00 
63.00 


5 

3 


total 


159. 






Undecided 


61.26 


39 











COLLEGE CREDITS COMPLETED 



0-15 credits 


61.12 a 


86 


16-30 credits 


61.30b 


23 


31-45 credits 


65.00 


13 


46-60 credits 


63.00 


* 12 


Over 60 credits 


69.62 ah 


26 


AGE 






Under 20 


60.88a 


92 


20-24 


65.53a • 


36 


25-29 


64.73 


11 


30-39 • 


67.54 


13 


Over- 39 


65.88 


8 



rou|f n 

groups 
total 



4 390.26 

155 63..61 
159 



6.14 .0001 



SroiP 
total 



155 
159 



252.50 
67.17 



3.76 .006 



GRADE POINT AVERAGE (4-pt. scale) 



Under 2.00 
2.00—2.49 
2.50—2.99 
3.00—3.49 
3.50—4.00 



S9.42 a 

63.39 

64.12 

67.50a 

65.90 



24 
23 
42 
28 
10 



een 



S&ujs 
total 



122 
126 



4 223.10 
59.96 



3.72 .007 



COMMUNICATION COURSES 

None 61.01 71 

1 63.12 49 

2' 64.58 19 

3 67.70 10 

4 66.83. 6 

5 or more* 69.40 5 



between' 
groups 



ip£ 

total 



5 841.89 
154" 68.69 
159 



2 ! .4S .036 



PAST SPEAKING EXPERIENCE 



61.06 ao 49 



None . 

Classes other than 

speech 60.35 c 

Speech Classes -62. 284 

Some Experience out- 
side of speech 
class 66.52 a 

Good amount out- 
side class 72.50jj Ct j 

A lot outside class 65.75 



26 
46 



25 

10 
4 



between 
groups 

within 
groups 



total 



154 



159 



326.68 5.14 .0002 



63.56 



♦Means with a common letter in the subscript differ at or beyond the .05 
level of confidence utilizing the Tjukey^procedure. 
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TABLE 4 

CORRELATIONS OF DEMOGRAPHIC VARIABLES AND THE 19- ITEM SHORT FORM 





cc 


Age 


GPA 


CCC 


SE 


Race Sex 


Credits Completed 














Age 


"34*** 












Grade Point Average 1 


.31*** 


.29** 




. 






Communication Courses 
Completed , 


.34*** 


.19* 


.11 








Speaking Experience 


.21** 


.15 


.06 


.58*** 




• 


2 

Race 


-.04 


-.02 


- 32*** 


-.06 


-.02 




Sex' 5 


-.08 


.18* 


.02 


.06 


.07 


-.01 


19- Item Short Form 


.35*** • 


.21** 


.28** 


.28*** 


.31*** 


-.27*** -.02 


*p < .05 c **p < .01 


***p 


< .001 








c 


*33 students had no GPA accumulated; they were 
correlations. For all other correlations, N 


labelled 
= 160. 


"missing" for this analysis. 


N « 127 for all GPA 


2 

For the Race variable, 


0 - Non -minority, 1 ■ 


Minority. 








For the Sex variable, C 


a Male, 1 = 


Female . 
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"TABLE 5 

PERCENTAGE OF STUDENTS' SCORES ON A S-POINT SCALE 
FOR THE 19-rTEM- SHORT FORM 



Item ' 


lowest** 
1 


2 


3 


4 


•highest 
5 


Pronun ciat ion 


1.2% 


«t 0.0% 


14.4% 


47.5% 


36.9% 


Facial Exnr~ /Tone o£ Voice 


0.0 


~ 9.4 


29.4 


41.2 


20.0 


• 


1.2'* 


15.0 


36.2 


43.1 


4.4 


rcrsUodlVv/ xiiXUiiii. uidcjjiv>i • 


5.0. 

W • V * 


20.6 


30.6- 


31.9 


11.9 


wlariLjr OX XUCao 


1.9 


12.5 


36.9 ' 


39.4 


9.4 


Exnwc« & Defend Pt. of View 


X.s 


. 27.5 


38.7 


18.1 


8.1 


Rftcoffni2e Non "Understanding 


3.1 


2.5 


23.7 


46.2 


24.4 


Fact /Od in ion Distinction 


4.4 


5.6 


23.7 


61.2 


5.0 


Listening f Understand Suggestions) 


si 


22.5 


47.5 


20.6 


4.4 


Identifv Main Ideas 


3.7 


. 10.0 


38.1 


28.7 


19.4 


Summarize < s 


8.1 


7.5 


36.9 


22.5 


25.0 


Social Ritual' 


1.9 


10.6 


30.0 


40.6 


16.9 


Ask Questions 


• 1.9 




&/ • O 


XA A 




Answer Questions 


1.2 


8.7 


55.6 


26.9 


- 7.5 


Express Feelings* 


5.6 


29.4 


19.4 


31.9 


13.7 


Organize Ideas 


20.6 


12.5 


39.4 


26.2 


1.2 


Give Directions 


12.5 


20 c 0 


30.0 


20.6 


16.9 


Describe Another's View 


17.5 , 


31.3 


28.7 


18.8 


3.7 


Describe Diff. in Opinion 


J 4 ' 4 


; 30.0 

i 


34.4 


16;9 


4.4 
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POTENTIAL SOURCES Of BIAS ZH SPBAIING AMD LISTENING ASSESSMENT 



Richard J. Stiggins 
Clearinghouse for Applied Performance Testing 



ABSTRACT 



There is some evidence to suggest that the um of performance 
assessment can, in certain educational assessment contexts, minimise the 
adverse impact of testing on minorities. Performance assessment, ah 
alternative to traditional paper and pencil testing, relies on the direct 
observation and rating of behavior by qualified judges. * For obvious 
reasons, such assessment plays a key rola-in the measurement of speaking 
and listening skills. Because patterns of oral communication are greatly 
affected by potentially divergent linguistic and cultural factors, great 
care must be taken to deal with potential sources of bias in the 
assessment if .valid and reliable results are to be achieved. This paper 
reviews those potential sources of bias, and suggests strategies for 
dealing with each. Purposes and strategies for assessing speaking and 
listening skills are reviewed, as are the characteristics of good 
speaking and listening tests. Specific sources of bias in speaking and * 
listening assessment are identified in (a) test administration 
environment, (b) stimulus conditions, (c) response modes, and (d) rating 
or scoring procedures* Specific suggestions are offered lot planning and 
conducting a speaking and listening assessment that precludes problems of 
bias. v 
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FOTBfTCAL SOURCES Of BIAS ZK 

SBSftkna ahd lzstsnihq asssssmbit 



A teat is rigirM y biased when mm characteristic of the test 
interacts with mm char actaria tic of trie-, teat takar la auch a nay aa to 
distort the meaning of tha raaulta for a oar tain group of examinees 
(Shepexd, 1980) . 8uoh distortions can laad to Invalid assassMnts and 
inappropriate educational decisions. Foe example* taat items uaa 
language or. syntax more readily comprehensible to sMbars of one group— 
with its cultural and linguistic experience— than to anbthar group. Or 
tha axarcisas may ba baaad on problem situations with which ona group has 
simply had more familiarity than another.** In both cases, axaainaas from 
aach group sight ba squally capable of performing competently on tha 
test, but aoM language or experience factor extraneous to tha trait or 
skill baing measured night distort taat raaulta and thus laad to an 
invalid assessment. 

Shis papar axploraa some of tha potantial sources of biaa in spaaking 
and liataning aeseaament. Bacausa apaaking ak ilia and liataning skills 
call for diffarant MaauraMnt strategies, thay ara traatad separately. 
Bach if daacribad in tans of assessment purpose and strategies, 
characteristics of a good taat, potantial sourcaa of biaa, and atratagiaa 
for avoiding biaa. 



Assessing Spaaking Skills 

Purpoaa and Strategies . The purpoaa hera ia to Masure tha 
examinee's ability to oommunicate orally, Assessors have two measurement 
optional (1) a papar and pencil objactive taat measuring knowledge of 
tha principles of good communication, or (2) direct observation of 
examinees' ability to apply those principles in a real speaking context. 

Although knowladga of effective coamunication principles may 
correlate highly with actual apaaking^ proficiency, seat wdbld contend 
that direct observation of an examinee's performance yields Mra 
depandabla conclusions regarding spaaking proficiency. In short, the 
Mrs effective strategy is to alicit a sample of speech in a real or 
simulated actuation, and to ask a trained ratar to evaluate results 
according to prespacified perfontanoa criteria. Spaaking proficiency can 
be rated holistically or it can ba subdivided into such components as 
organisation, content, and expressiveness, and ratad analytically— 
depending on tha assessMnt purpoaa or tha informatioc dasirad. 

Character iatics of a Good Taat . Teat characteristics necessarily 
vary somewhat as a function of teating purpose. All effective speaking 
teats, however, share mm common characteristics! (1) clearly focused, 
carefully prepared speaking exercises; (2) multiple realistic (real or 
simulated) apaaking contexts; (3) clearly specified (in writing) 
performance criteria; (4) effective rater trainings and (5) a carefully 
conducted rating process. 

Tasting purpose determines the level of information required in test 
results. For exempli, placement into remedial speech courses typically 
calls for information on overall communication proficiency. On the other 
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hand, diagnosis of strengths and weaknesses for purposss of assigning 
individualised instruction requires thorough analysis and scoring of ths ' 

component skills of oral communication. 

Potential Sour ess 4 of Bias . Thsrs art at Itast four pottntial sourcas 
of bias in a parformanoe tast of spaaking skills tha tast administration 
context, tha stimulus situation, tha rasponsa mode, and tha scoring 
prooess. 

Bias oan com from tha tast administration environment, If that 
environ men t is mora familiar or oomfortabla to one cultural group than 
another. Xnvironmental factors that can affact performance include 
amount of anxiety with tha tast format, satisfaction with physical 
surroundings, fraquancy and haturt of distractions, and race or sax of 
tha tast administrator. Khan thtsa factors affact ona group 9 s scores 
mora than anothar 9 s, thay can laad to invalid conclusions regarding 
spaaking skill. 

The exercises (or stimulus) exercises can also ba biasad if based on 
inappropriately narrow cultural perspectives. Competent examinees who 
lack understanding of or experience with item content may be 
inappropriately judged incompetent. 

Response bias can arise as a result of cultural differences in the 
nature and form of oral communication typically used by examinees. Por 
example, members of -certain cultural subgroups' may be uncomfortable with 
formal speaking contexts. Though competent speakers, such persons aay 
experience sufficient anxiety or lack of motivation in an unfamiliar 
situation, or one perceived by their culture as inappropriate, to distort 
test results. 

And finally, bias can arise in scoring* whenever raters are called on 
to make subjective judgments. The potential for rater bias is a function 
of the raster 9 4 social experience and attitudes, the clarity and precision 
of the scoring criteria and standards/ and the extent to which the rater 
has internalised those standards. Guilford (1954) has outlined tiie many 
forms of bias in ratings, and others (Sdhmitt and Lappin, 1980s Campbell, 
1972; Quinn, 1969* and Brignoli, at *1., 1979) have shown how these 
factors influence performance ratings of different cultural subgroups. 
Bater bias is perhaps the most significant threat to test score 
dependability in performance assessment. 

Avoiding Bias . The general strategy for avoiding bias in speaking 
assessment i* to eliminate those factors that can 4 distort the meaning of 
test scores. Tor example, the assessment can be administered in a 
familiar and comfortable environment by. & test administrator who is not 
likely to be intimidating or otherwise distracting to examinees. 
Stimulus exercises can be reviewed in advance by members of appropriate 
cultural subgroups to ensure relevance and familiarity to all examinees. 
The use of forma], and. informal speaking exercises which are within the " 
real-world experience of all examinees can redude .response bias and 
anxiety, and increase motivation to perform. Another strategy for 
controlling anxiety is unobtrusively assessing speaking skills via direct 
observation" of examinees in everyday communication settings. And 
finally, rater bias can be controlled through the very careful 
specification and review of performance criteria/ the careful selection 
of raters and thorough rater training. 
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AiMMlnq Listening Skills 

Purpose and Strategies. The purpoM here is to measure an examinee's 
ability to comprehend oral communication. As with speaking assessment, 
tba examiner has two option*! a tast that measures knowledge of the 
principles of good listening, or a performance test that measures ability 
to apply those principles. 

The wet effective strategy is typically to use a performance test 
that relies in part on objectively scored tast items of listening 
comprehension. That it, the listening exercise is presented orally in a 
real or simulated context, and the. examinee 9 s comprehension of the 
massage la tested uaing traditiona l multiple choice items. As an 
alternative to written items, one can ask the examinee to paraphrase all 
or" part of the message, and rate the level of comprehension ooweyc. in 
that paraphrase according to prespecif ied criteria. 

Characteristics of a Good Teat . A good listening test is one that 
provides the kind of information needed for appropriate educational 
decisions. Head (1978) has pointed out that such a tast focuses on 
skills central to the listening process by using real verbal * 
communication presented orally, rather than uaing reading test passages. 
Head urges the use of short, interesting stimulus materials that are 
relevant to the communication context and experience of the examinees. 
In addition, Head contends that a good test should not confound listening 
and reading *kill by requiring that examinees read teat items after 
listening to passages. Instead, test items should be presented in both 
an oral and written form. And finally, Head suggests the use of items 
that require (1) recalling significant details, (2) comprehending main 
ideas, (3) .drawing inferences, (4) making judgments about speakers, and 
(5) making judgments concerning information presented. 

Raters who are asked to judge listening prof iciency based on an 
examinee 1 s paraphrasing skill must be carefully trained and must have 
clearly specified performance criteria on which to base their judgment. 

. Potential Sources of Bias . As with speaking arseesment, sources of 
bias in listening assessment can include the test administration 
environment, stimulus, response mode and scoring procedures. If the test 
administration situation is unfamiliar or uncomfortable to certain 
cultural groups (more than others) or tte r Mmulus materials are based on 
narrow cultural or linguistic experience* o , perspectives not common to 
all examinees, the test can yield biased f.sw.s. 

The response mode can affect biar ~ / objective test items are 

used to measure comprehension,, such fa.' vxv. differences in test 
wiseness, anxiety or motivation to do "w^i on Isad to biased judgments. 
If paraphrasing is used to measure comprew.tslow speaking and listening 
proficiency *ay be confounded. Cultural difference* in speaking style 
say also lead to biased assessment. Whenever factors extraneous to the 
actual listening ability of examinees influence test scores, and do so to 
differing extents across cultural subgroups, the meaning of test scores 
will be distorted for members of some groups and the test scores will 
lead to biased judgments about listening proficiency. 

And finally, scoring procedures can contribute to test bias when 
subjective ratings of paraphrased messages are the basics for determining 
proficiency. Rater bias is, of course, not a problem with objective test 
items. 
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Avoiding Bias. The key to avoiding bias in listening assessment is 
to use familiar, realistic and relevant .samples of oral communication 
combined with high quality assessment procedures. Careful review of 
stimulus materials, items and rating criteria by members of appropriate 
cultural subgroups prior bo test administration can increase quality and 
help eliminate bias. Test items should J* prepared by qualified testing 
specialists, and subjected to careful pretesting, and psychometric ' 
axulysis jrior to actual administration. Bias in the response mode 
betause of differences in anxiety and motivation can be reduced by using 
paraphrasing as an alternative to the traditional objective test. These 
gain** however, can be offset by rater bias unless raters are thoroughly 
trained and given carefully specified criteria on which to base their 
judgments. 



Conclusion \ 

In speaking and listening assessment, test environment, stimulus 
conditions, response mode,* and scoring procedures all influence the 
meaning and utility of test results. If performance tests are carefully 
developed by persons knowledgable about and sensitive to the cultural, and 
social experiences ^d traditions of the groups to be tested, such tests 
can provide useful, fcearingful scores. The use of stimulus circumstances 
and exercises e^ually^amiliar to different groups can help ensure that 
everyone has a chance to perform successfully. When members of different 
groups are faced with a test stimulus or response mode that prevents them 
from demonstrating their ability , the test opportunity is biased. The 
\jse of performance rating procedures not based on explicit criteria, and 
not applied by carefully trained judges, can lead to rater bias. 
Considerable care must be taken to prevent both opportunity and rater 
bias* since either can significantly influence the dependability of 
speaking and listening assessment results. 
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USING PERFORMANCE RASING SCALES IN LARGE SCALE 
ASSESSMENTS OP OPAL COHMONICAXION -PROFICIENCY 

Donald L« Rubin 
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ABSTRACT 



Speech performance rating procedures 'optimize both the 
inf ormativeness and the pedagogical "washback" of communication f 
competency assessments. .Rating procedures can be adapted to a variety of 
communication situations, and may serve a number of administrative 
purposes. Appropriate selection of rating instruments and criteria is 
determined by the performance situation and the purpose of the test. 
Considerations of content validity beat on choices of performance 
situations as well as rating criteria* Considerations of reliability 
bear on the size, number , and variety of performances sampled as well as 
on agreement between raters. Considerations of < feasibility bear on the 
time and personnel, needed to elicit and rate speech performances. 
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USXHG SKRFOitlfMKS RATING SCALES ZH LARGE SCALE 
ASSESSMENTS OF ORAL C0MtJNIC*TION PROFICIENCY 



Federal legislation presently includes oral communication in its 
enumeration of basic skills (ESSA Fart 162, Federal Register , April, 
1979, 25151)* Many state and local education agencies have already begun 
to test oral communication 'competencies, while others have initiated 
preliminary planning in .this direction (Backlund, 1981) . This paper does 
mt argue: for large scale testing of oral communication proficiency. 
Indeed, it is preferable not to test when an assessment program threatens 
to undermine instruction* This paper is, however, promised on the fact 
that programs for testing oral communication competencies are upen us. 
It is therefore critical to examine available assessment methods in order 
to select those which will maximise the benefits of such programs. 

Great Britain, where large scale testing of oral skills has been 
conducted for some time, provides a valuable example of the impact of 
oral assessment on education. ' British Certificate of Secondary Education 
examinations have included some type of speech sample since 1965 
(Bitchman, 1966; Schools Council, 1966). Although curricular documents 
have stressed the importance of •oracy* (Bullock, 1975), British 
educators recognise that the institution of formalized testing lent 
respectability to speech instruction (Wade, 1978) and shaped its nature 
(Wilkinson, 1968). Barnes (1980, p. 125) observes that *any monitoring 
of oracy during secondary schooling will be proposing a wider range of 
curricular concerns jin oracy than schools presently undertake. • .Thus, in 
secondary schools at least, the monitoring of oracy is likely to be 
leading practice in schools rather Xh*n responding to it.* 

With an eye toward effects ofv pedagogical as well as psychometric 
adequacy, then, this paper examines one type of communication assessment 
procedures speed! performance rating* It constructs a rationale for 
this procedure and describes conditions influencing selection of 
particular speech performance rating techniques. The papar also 
addresses the manner in which considerations of content validity, 
reliability, and feasibility can affect the adequacy of speech 
performance rating as it is likely to be implemented in school settings. 

Rationale for Speech Performance Hating 

Speech performance rating ^encompasses assessment procedure? in which 
student speech is sampled in some more or less communicative context. 
Judges rate the speech samples according to some criteria of 
communication quality* Speech performance rating thus excludes more 
narrowly directed testing of linguistic ability x for example, tests of 
articulation requiring students to read word lists, or oral tests of 
grasmar which employ criteria making reference only to syntactic 
patterns. More to the point of this essay, speech performance testing 
.contrasts with indirect tests of speaking proficiency wherein students 
are questioned about oral communication but do not engage in it.. 
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Indirect test* are adventageoue btoiuM they art easily administered 
and aaaily eoored. Though indirect taata of akiU auraly lack fact 
validity r they ^ay indeed ba highly valid indicatora or predictora of 
ooapetenoe to parf on. . As an example, oonaidar multiple-choice" taata of 
writing ability (Ireland, 1977f Oodahalk, Bwineford ft Coffman, 1966| 
Palmer, 1961). „ Speaking ooapetenoe, however* haa proved reeiitant to 
indirect tatting. Ona notable aff ort met with difficulty in constructing 
adequate items (Mead, 1977) • Communicative intentione ara ocaplex and 
ooamuaicative contexts are wultidiaeneional. Naaning can sometimes ba 
oonveyod af f actively by deviating f roa norma. ' for example, a apaakar 
can* under cartain circumstances, express endearment by uttering* "You're 
•uch a }erk, Jackson.* Competent apaakera can anviaibn oonditiona undar 
Which naarly any auppliad alternative could appropriataly aerve aoaa 
ooaaunlcative intention, and it ia no aaan taak to sufficiently 
circuasaribe a ocaaunicativa eituatioh within tha oonfinaa of an 
examination question. Indaad, Howie-Day (1977) found no age-dependent 
patterning in choioa of auppliad persuasive appeala, but only in # & 
rationales used to explain thoaa choices* 

Indirect taata of speaking ability rapraaant a challanga to taat 
oonatruotion, but aore importantly , thay alao rapraaant a threat to 
acceptable inatruction in oral ooanunl cation* Wilkineon (1968) daacribad 
tha manner, in j*hich tha aoda of oral aaaaaaaant haa a "waahback" aff act 
on tha manner in which oracy ia taught* Similarly, Aubin (1980) 
diacuaaad tha "padagogical validity* of oral aaaaaaaant procedures which 
ia ascertained by determining what qhangee in (a) curricular contant and 
(b)~ instructional practical would maximize teat scores • Zf indirect 
tasting of apaaking ability wara inatitutad for decisions of major 
oonaeqtience, it ia likaly that atudarita would accordingly aoon ba 
learning about ooanunication instead of learning how to communicate* The 
experiential oaaponent which oonatitutea tha ooty of moat communication 
training would/ atrophy* Tha hiatory of Educational Taating Service' 8 
,Collega Board English Composition Teat provides an instructive parallel* 
Taat constructors developed a reliable and coat-effective instrument 
which predicted writing sample ratings with great accuracy* Original 
writing samples .ware consequently eliminated fro* the College Board 
Composition Teat* However* English teachers protested that the indiract 
teat cauaad a reduction in writing tlae-pn-taak and an increase in 
punctuation and uaage drill* Ultimately this pressure resulted in 
reinstitution of soma type of .writing sample (Educational Testing 
8ervice, 1979). 

Tha limita of teat construction and .the demands of acceptable 
instruction* then, poinc to tha need for direct performance testing of 
speaking akiU. Still* it may be ^rgued, rating procedures may not be 
the moat efficient means of assessing quality from speech Samples. The 
*>st viable alternative to rating ia some form of discourse feature 
analysis. p That ia* apaaking akill may be inferred on the basis of 
specific structures, strategies, cr quantifiable itema appearing in 
speech samples. 

Both Loban (1976) and MoCaleb (1979) suggest syntactic complexity as 
■a discourse feature from which evaluations of communication competence . 
can be drawn* But while ayntactic complexity ia generally an index of 
ayntactic maturity* it is alao affected by a variety of contextual 
factors/ and in nc way directly reflects quality of expression 
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(Crowhurst, 1979) • Indeed* in writing examinations, tht most able 
writers say produce the least complex syntax (Rosen, 1969) • In a like „ 
vein, students who participate in instruction in referential 
communication accuracy often produce briefer, denser messages (Fry, 1969; 
Shants a Wilson, 1972). \ But no one would consider such telegraphic 
speech to be qualitatively superior communication. Context-creating 
statements were taken am*fa reflection of audience adaptedness in one 
study (Braoewell, Scardamalia * Bereiter, 1978), but were shown unrelated 
to age or expertise in another (Rubin ft Piche, 1979) . In short, speech 
is a woodrously varied and flexible tool/ and any equation absolutely 
linking specific discourse features with quality of expression is doomed 
to disoonf irmation. 

-Speech performance rating— cumbersome, liable to measurement error, 
relatively costly— appears to be the optimal method for assessing 
speaking proficiency. It has no inherent impediments to validity* it is 
clearly relevant to the target skill domain. Moreover, speech 
performance rating may have a salubrious effect on instruction. Though 
challenging, it is feasible; its technology is currently available. 



Situations for Performance Rating 

An emphasis on speech situations can be misconstrued as an emphasis 
on speech types* the persuasive speech, the panel discussion, the 
dramatic reading. This interpretation is misleading because it 
oversimplifies the dynamics of communication behavior and can lead to 
mechanical, rote preparation and performance, A more adequate framework 
refers to communication functions realized in the context of particular 
audiences, particular tasks or topics, and particular settings.. We 
communicate to inform, to express our emotional state, to regulate 
others, to persuade, to bond with others, and so on (Allen ft Brown, 1976; 
Jacob son, I960). (And interactions generally serve several of these 
, functions simultaneously) . Horeover, we execute these functions 
differently under varying circumstances, A formal presentation about 
competency testing, for example, will be adapted in a number of respects 
If it is being delivered to a school faculty as compared to a civic 
group, and adapted again differently if delivered from behind a podium as 
compared to from a seat at a round tabled 

Accordingly, evaluation criteria need to reflect the peculiar demands 
of speech situations. It is not reasonable to judge, say, an interview 
performance by means of the same criteria as are applicable to a peer 
discussion. Similarly, an informative speech calls for standards 
different from those for a ritual speech of appreciation. By the same 
token, genteel pronunciation and usage typical of dominant cultural K 
groups may' be appropriate in some situations, but not in others* 

Rating procedures can account for situationally appropriate 
adaptation in several ways. Some scales include* an item explicitly 
citing appropriateness or adjustment to audience and purpose. In another 
approach, the rubrics describing levels of quality will be different for 
different speech situations. For example, a high rating along a 
dimension of "intelligibility* may require avoidance of* technical jargon 
when addressing a heterogeneous audience, but allow specialized 
vocabulary when speech is directed to a specialized audience. Finally, 
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different speech situations say require th* application of altogether 
different criteria, reflecting divergent rhatorical demands of thosa 
situations (Lloyd-Jonas, 1977) . Thus, an item "shows initiative" may ba ' 
included in a seals for rating simulated job interviews, while an item 
"refutes counter-arguments" would ba fitting only in evaluating a 
persuasive parformanca. p 

Spaach situations can ba arrayad along a continuum frosi formal, 
axtended, uninterrupted, and plannad on tha ona hand to informal, 
raciprocal, spontanaous and nendiractad on tha othar (Wilkinson, 1968) * 
Nora formal spaach situations yiald graatar control over tha examination 
and graatar aasa of rating* Thasa situations may not ba of tha highest 
life-role utility, however . In addition, formal speech situations may be 
foreign and threatening to large numbers of students, resulting in 
depressed levels of performance. These situations may be especially 
alien to members of cultural minorities (Gay t Abrahams, 1973; Philips, 
1970). 

Because performance appropriately varies as a function of situation, 
and because different situations may tap divergent constellations of 
skills, an ideal assessment program in this domain would sample 
performance in a variety of situations. This paper returns to the 
problem of sampling spetch situations in a later section on content 
validity, and addresses the question of equivalence of performance across 
situations in a section on reliability. Some administrative options 
providing for ^ variety of rpeech situations in, an assessment program .may 
be briefly mentioned at this juncture, however. 

Utilising a single speech situation obviously simplifies 
administration and reduces costs, in one program, initial tasting takes 
place in the context of a persuasive public address, while xetasting of 
those who fail to achieve criterion performance levels takes place in the 
context of a simulated job interview (Rubin £ Bazfcle, in preparation) • 
This arrangement requires that the reliability and difficulty of 
equivalent forms be verified. Alternatively, individual students may be 
observed only in a single speech situation (self -selected or randomly 
assigned) , with, that situation drawn from a list of testing option* 
(e.g., Gary Community Schools Corporation, 1977-1978). Again, 
equivalency among these options must be established. Finally, students 
may be subjected to multiple observations representing a range of 
situations. This procedure is administratively feasible only if it is 
conducted by regular classroom teachers and incorporated into classroom 
instruction, at least for initial screening (e.g., Massachusetts 
Department of Education, 1979; Vermont Department of Education, 1978) • 
Use of inadequately trained classroom teachers as raters, however, risks 
considerable unreliability. 

Reading Aloud; This most formal of speech situations affords great 
ease of administration and scoring. For this reason it is the most 
common situation employed in British Certificate of Secondary Education 
examinations (Barnes, 1980; Bitchman, 1966, 1969; Wilkinson, 1968). It 
should be noted that tests composed of reading aloud can reflect more 
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than ability In vocal (Salivary— articulation, pronunciation, and the 
like. Hitchman (1966) proposes that proaa and poatry reading ba judged 
by the criterion of "interpretation," which signifies akill at conveying 
to a listener the. mood and Meaning of a passage* 

Extended Honolocrue* Everyone has a notion of what it seam to give a 
speech. Unfortunately, the conditions for speech making— an authentic 
purpose and a^ potentially receptive aadience— are frequently absent in 
classroom practice, audi less in ccal competency testing. Hence the term 
"extended monologue." .Extended monologue assessments are most gdaquate 
when they make provision for at least a simulated audience and purpose. 
In one program, for example, students conduct a simulated public hearing 
before a Board of Education. Bach mtudent makes a presentation before 
the Board, taking a position on one of three agenda items (Rubin a 
Bessie, in preparation) . students seem to perform better when they are 
dealing with personally meaningful subject matter in a well-defined 
situation. Instructions that read *Speak for three minutes on the proper 
role of American foreign aid"* are not likely to elicit maximal 
performances from most youngsters. 

; Group Discussion . Group discussion formats spar* a range of speech 
situations from panel presentations— which may be highly formal—through 
free discussion. Group discussion tasks deserve consideration in oral 
assessment programs first, because peer interaction may optimize some 
students 9 performance; second, because small-group problem solving has 
value as an instructional technique across the curriculum; and third/ 
because small-group communication skills are of high utility in life-role 
contexts. Group discussion, assessment tasks entail some especially 
thorny challenges to measurement, however (K*ltner, 1953). The quality 
of individual discussants 9 performance is affected by the quality of 
other members 9 performances. Different group roles— leader, for 
instance— require the . application of different evaluation criteria, when 
raters take active roles in directing discussion, they can alter the 
quality of students 9 performances in ways which are extraneous to 
students 9 communication skills (Barnes, 1980) . Still, it appears that 
discussion skills can be rated in a reliable manner (Becker, 1956). 
Criteria mentioned in conjunction with assessment of group communication 
performance emphasize cooperative thinking', integration into the group, 
and appropriate degree of participation (neither reticent nor dominating) 
(Becker, 1956; Crowell, 1953). 

Interview . Bazen (1978) rightly observes that under conditions of 
testing, dyadic communication never qualifies as true conversation. When * 
a rater interacts with an individual testes, the situation is structured 
and goal oriented; it is '&n interview. Some assessment procedures 
attempt to circumvent this state of affairs by simulating conversational 
performance through role-playing (Massachusetts Department of Education, 
1979).. Others engage students in direct interviews about 
autobiographical information (Gary Community School Corporation, 
1977-1978) • Simulated job interviews have also been used for oral 
assessment (Rubin a Bazzle, in preparation) • Tests of second language 
proficiency (Mullen, 1978) concur with experience in testing 
communication proficiency (Bazen, 1978; Hitchman, 1966) in suggesting 
that the interviewer/rater can seriously affect students 9 performance 
scores. 
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jguggoggg for Speech gerfgniance Rating 



Intonation derived from larga acala parfotmanca assessment may serve 
aa input foe Several typaa of educational daciaiona. Taat results may ba 
uaad to oartify tba competency of individual students for purposea of 
promotion or graduation. Tba results say ba uaad to place students in 
specific instructional sequences in order to remediate deficiencies or 
enrich existing strengths. Or, assessment information can ba uaed to 
evaluate the effectiveness of programs or of teachere. 

Tba intended purpbee of a speech proficiency taat will determine, in 
part, the type of rating acala needed. Instruments designed for the 
purpose of certifying communication competency must somehow reflect those 
oral competencies previously identified by educational agencies* This 
can ba acoonpliehed most satisfactorily by conatructing acala items 
corresponding to each competency. For complex competencies which have 
not been defined with preciaion {e.g., "Deliver a siopla speech to inform 
about a personal interest or hobby") it may ba naceaaary to construct 
scale items corresponding to sOSakills underlying the mora general 
objactivaa ("uses intr induction," "fotmulatae a logical sequence of 
ideas," "oral delivery is intelligible," etc.). If the resultant rating 
acala ia deemed too unwieldy, it is possible to create, a simpler, more 
impressionistic scale composed of fewer items. For such simplified 
scalee, rubrica describing indicators for each level of quality will 
expreaa the same competencies or; subskills as would appear on a more 
complex instrument. The simplified instrument may seem ostensibly 
uninforfcative* students must be given an explicit list of rating criteria 
in order to adequately prepare and to interpret their results. 

When test results are to be used for the purpose of student diagnosis 
and placement, there ia little alternative to a highly detailed rating 
inatrusent. Scale items will explicitly state evaluative criteria 
corresponding to instructional goals. Thus, a student who rates poorly 
on "ideas sufficiently elaborated" may participate in an instructional 
module training invention skills through practice in use of heuristics, 
brainstorming, and use of forms of support. That same student may not 
require the additional training in organization skills or in kinesic 
delivery skills which may be indicated for another student. 

The effectiveness of a program or of a teacher is generally 
ascertained by assessing the product of that program or teachers student 
achievement (though this approach is useful only when student aptitude 
and institutional resources are also taken into account). Rating scales 
for use in summative evaluation of achievement need not explicitly state 
detailed criteria if the ultimate users of the test results are simply 
interested in documenting instructional impact. More detailed 
information will be required if test users are interested in "fine 
tuning" existing programs. Of course ratera will need to be well versed 
in the specific evaluation criteria, in any event, if they are to judge 
students 1 oral performances in a consistent fashion. 



Types of Speech Performance Rating Scales 

It is not possible to recommend a single type of rating scale for use 
in speech proficiency assessment. Instruments vary in the degree to 
which items represent detailed competencies or subskills. They differ 
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also in the degree to which criteria are .tailored to particular demands 
of communication tasks. In order" to select .an appropriate rating scale, 
testing agencies must consider the purpose of the assessment program and 
the speech situations in which students 9 oral performance are sampled. 

Typ*-of-8psech Checklist . ~ Some speech assessment programs seek to 
inoorpot ^> evaluation into normal classroom activity. This approach may 
mitigate student anxiety and also relieve* (or shift) some administrative 
burdens. Also, it allows for testing in a 'variety of speech situations. 
The major disadvantage of such schemes is lack of control over 
administration and rating, with attendant lack of reliability in 
measurement. One way in which oral proficiency testing can be 
assimilated into classroom activities is to require students to 
participate in a number of speech activities, such as delivering a 
simulated sales pitch, reading a poem aloud, giving directions on finding 
a local landmark, and the like. , Teachers simply check off for each 
student the activities that jthe student has "successfully accomplished 
(Vermont Department of Education, 1979; Pennsylvania Department of 
Education, 1980) • Beyond problems of establishing reliability across 
different teacher/raters., and activities/administrations, type-of - speec h 
checklists may beg the issue of what, is to count as a successful speech 
performance. In order for this type of evaluation to yield neaningful 
information, each speech activity must be subjected to detailed task 
analysis and the qualities ;of mastery-level performance carefully 
described. ^ t 

General Impression Marking ." Rather than analyzing a performance 
atomistically, general impression, marking assigns a single global score 
representing a qestalt sense of quality (Cooper, 1977; Gods chalk, 
Swinef ord a Ccffmari, 1S66) • Generally', raters assign general impression 
marks quickly, based on immediate, reactions. . Scales usually comprise a 
five-to-ten interval Lickcrt type metric ranging from * very poor 9 to 
"very good." Evidence suggests that seven-step scales are optimal 
(Becker a Cronkhite*, 1965). With extensive training, raters can render 
highly reliable ratings on general impression scales. Use of these 
devices can also be supported by detailed -verbal descriptions. 
(Thompson, 1944) or by videotaped examples (Bowers; 1964) representing 
typical performance, at each level of quality. 

Analytic Scales . Analytic scales, widely. used in classroom and 
contest evaluation of speech, consist of more than one item and require 
raters to judge separate dimensions of speech performance. Each item 
represents a criterion like "organization," "supporting material," 
"language flpencry," and "voice quality!" Bach criterion is rated on a 
multi-interval scale, much as in general impression making. The 
selection of criteria/items 'is* a major problem of content validity. 
Becker (1962) suggests that typical analytic scales reflect only three 
clusters of judgments* content analysis, ■ delivery, and language. As 
previously mentioned, some analytic scales attempt to incorporate the 
notion of context-appropriate adaptation by including an item like 
"speaker adjustment." Frequently, too, analytic scales include a 
"general merit" item. 
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Dichotoaous Choice Scales , it is generally eaaier to discriminate 
tht presence or Absence of a trait than to discriminate between aavaral ' 
qualitative levels of that trait. Consequently, apeach rating scales 
hive been developed which ask ratara to render f oread choice on aach of 
aavaral iteas (Brooks, 1957) . Thasa ara essentially a variant of 
analytic aoalas wherein aach criterion is avaluatad by Mans of a 
two-interval item. Construction of dichotomous choica scales r squires 
the same analysis of the content/skill domain aa does construction of 
analytic acales. 

Rhetorical Trait Scales . Primary tgcait scoring, introduced by Lloyd 
Jonas (1977) for use in evaluating Writing ability, takes as its starting 
point a rhetorical analysis of the demands peculiar to the communication 
teat situation. The scat important, or "primary" rhetorical traits 
associated with the communication task become items which are scored in 
accordance with very specific indicators of qualitative levels. 
Rhetorical trait scales for use in asseaaing speaking proficiency operate 
similarly, except that they may include more items or traits, not seeking 
to distinguish "primary" traite. A separate rhetorical trait seal* must 
be constructed for each aaaecament aituation, thus recognising the 
context-dependency of communication competence; The scales oonvey a high 
information load, and because of the Specificity of quality indicators, - 
they are eaaily adapted to competency based testing programs. Rubin and 
Baszle have developed one rhetorical trait scale (in preparation) for 
rating performance in a simulated job interview and another for rating 
performance in a ' simulated public hearing. One item from the latter 
scale, by way of illustration, appears as follows: 

Conclusion to presentation: (simulated public hearing) 
Suggested rating scale - 



Content Validity 

In discussing speech situational the paper has already addressed one 
aspect of content validity: how shall the domain of communication 
contexts be sampled? Taxonomies of communication roles, functions, and 
settings abound. One category system that has received repeated 
attention was expounded by Joos (1961) and includes (1) intimate, 
(2) caaual, (3) consultative, (4) formal and (5) frozen. Testing 
agencies might consider sampling situations from each of these classes. 
Zn any event, it ia readily apparent that, despite the ease of rating 
extended formal discourse, assessment programs which include only public 
speaking situations have not fully sampled this aspect of the content 
domain. 

Like selection of the speech situation, selection of rating criteria 
bear a on the content validity of speech performance tests. Lists of oral 
communication competencies have been promulgated (Allen a Brown, 1976* 
Basse tt, fflhittington 6 Staton-Spicer, 1978), as have various conceptual 



(1) 
(2) 
(3) 
(4) 



no conclusion or merely states that remarks are finished 
just thanks Commissioners or just restates position 
restates position and offers thanks 
summarizes or concludes memorably and offers thanks 
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■eh—* foe analysing components of communication competence (Niemann a 
Backlund, 1980 1 Wood, 1977)* Tha lack of consensus in identifying K 
qualities inherent in communication competence posas a graat barrier to 
establishing oontant validity of rating criteria? tha content domain 
remains largely undefined (Laraon, 1978). 

Surely, however, tha par m tar a of oral communication competence 
axtand bayond tha ability to uttar wall formed language. That ia, oral 
prof iciancy cannot ba equated with ganaral verbal ability, with Standard 
tngliah ayntax ;and pronunciation, or with "genteel" usage. These factors 
may oontributa to overall speaking ability. But oral prof iciancy also 
includaa production of nonverbal cuaa, control ovar extended patterns of 
coherence and organisation, invention of subject matter, contextually 
appropriate linguistic and strategic adaptation, anticipation of audience 
predispositions and ongoing sensitivity to audience responses, management 
of the mechanics of interaction ~uch as turn-taking, and so forth. 
Assessment scales which comprise criteria refering exclusively to more 
narrow language abilities lack content validity. 

'i 

Reliability 

,> 

Fair and meaningful performance retina requires consistent judgment 
and minimal measurement error. In past decades, speech communication 
researchers devoted substantial attention to reliability of ratings, for 
purposes of research, classroom evaluation, and contest judging. More 
recent concern about tha reliability of performance ratings in 
large-scale assessment programs has emerged from evaluations of writing 
ability. In general, research into the reliability of oral assessment is 
in need of renewed, attention. . 

Intra-Speaker . Students' performances are affected by such factors 
as time of day, environmental conditions, and physical health— factors 
extraneous to underlying skill. * Bow many speech samples should be 
elicited f ram each individual in order to obtain an .accurate estimate of 
oral proficiency? a single study bearing ot this question (Marine, 1965) 
found a teliabilit coefficient of about .78 for a single speech 
performance, incre ng to .91 for the reliability of a score based on 
the average of three performances. Based on this investigation, a single 
speech sample may be of adequate' reliability. In contrast, it is 
generally deemed advisable to rate at least two writing samples in tests 
of composition skill (Diederich, 1974) • 

Equivalent Forms . The nature of the performance task can affect 
student test scores in two ways. Zf students are given a choice of 
topic, then their performance may be affected by difficulties inherent in 
the subject matter. Rosen (1969), for example, found that in writing 
examinations students were often "penalized" for choosing certain topics, 
and that these were topics most often selected by more able students. 
Rubin and Bazsle (in preparation), however, found no significant 
differences attributable to choice of topic from among three persuasive 
propositions in a speech assessment. The issue of equivalent forms 
reliability is also raised when an oral examination program includes 
performance tasks representing different speech situations. Is a 
student's performance in telling a personal narrative, for example, 
predictive of that same student's score in group discussion? In one 
testing program, Rubin and Bazzle (in preparation) found that performance 
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in a simulated job interview correlated .72 with performance ratings 
derived from a persuasive speech delivered in the context of a simulated 
public hearing* *L 

Inter-Rater * Speech performance ratings inevitably involve 
subjective judgments of appropriateness and quality. Moreover, 
impressions due to extraneous speaker, qualities like physical * 
attractiveness may easily contaminate performance ratings. Under poor 
conditions, rater reliability may. range between .50 and .60 (Clevenger, 
1963; Marine, 1965) . Under proper circumstances it should be possible to 
achieve inter-rater reliabilities close to .90 (Applebaum, 1974). Since 
inter-rater reliability is partly a function of the number of judges 
(Clevenger, 1962). it is always advisable to employ at least two judges 
for etch performance. In one criterion referenced measurement effort, 15 
percent of the students were passed by one rater but not by the other 
(Rubin a Bazzle, in preparation) . When cross classifications occurred, a 
third judge *aa called in to resolve the certification decision. 

Several factors can enhance inter-rat«fr reliability. Among these is 
the nature of the scales on which ratings are made. Clevenger (1^963, 
1964) reported that less complex "general merit" items induced the 
highest reliability, though other researchers dispute this finding 
(Applebaum, 1974? Thompson, 1944). Reliability is increased when raters 
are encouraged to utilise all intervals on a ac*le, including the extreme 
poles, rather than marking only the intermediate levels of quality 
(Becker a Cronkhite, 1965). 

The conditions under which rating takes place can also affect 
reliability. Agreement may suffer as a result of rater, fatigue. Also, 
when raters must scote a speech after a subsequent student speech has 
already begun, reliability is not affected, but scores are elevated 
(Barker, Xibler a Hunter, 1968). Some evidence suggests that raters 
attain higher consistency when working with videotaped performances J.n 
relatively unpressured circumstances, rather than rating "live" 
performances (Rubin a Bazzle, in preparation). 

Raters are, of course, affected by their personality structures 
(Bostrcm, 1964), and their ratings 'of students 1 oral performances may be 
influenced by their attitudes towards thev speakers' subject matter 
(Miller, 1964). Judges differ in their leniency, trait, and hale errors, 
and it may be advisable to employ statistical controls to eliminate these 
sources of variation. (Bock, 1972) . 

Raters of similar experience and background will naturally respond to 
speech performances more consistently (Clevenger, 1963) <> . Bowers (1964) 
found that training of raters affected the absolute values of scores but 
not reliability. Experience most often demonstrates, however, that rater 
training can mitigate the effects of rater diversity (e.g., Clevenger, 
1963). Effective rater training familiarizes judges with rating criteria 
anjS rationales. Ideally, raters should have a hand in the initial 
formulation of assessment instruments. When raters are exposed to speech 
samples and share their scores publicly, they can calibrate themselves as 
a group and come to apply scoring criteria in a consistent and parallel 
fashion. It is helpful if raters are given the opportunity to 
periodically recalibrate themselves during the course of rating sessions. 
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Probably the greatest impediment to adoption of performance tasting 
in general, and speech parformanoa rating in particular, is 
administrators' ooncarna about tha poor coat-effectiveness of thesa 
procedures. Littla hard data ara available, to aithar corroborate or 
allay thaaa ooncarna. In ona comparison of tha financial coats of diract 
and indiract tasting of writing ability, it was found that tha coat par 
studant of conducting and rating a writing sample was $2.85, whila tha 
coat par studant of a locally davalopad standardised multiple choice taat 
was $8.06' (Hudson 6 Veal, 1980). 

Time is a major factpr in calculating allocation of institutional 
resources* She tine required of students and regular classroom teachers 
will vary aa a function of the apaech situation. Formal speech 
performance* for a class of 20 students may consume no more than a single 
hour-long class period if students are well prepared. On the other hand, 
individual interviews may take up to ten minutes per student with an 
equal amount of interviewer/administrator time required. Rater time muat 
slightly exceed the length of the .speech performance. The necessity of 
multiple raters for each speech performance greatly increases the time 
needed for oral performance assessment. One program employing two raters 
for each performance and a "floater" to resolve discrepancies used an 
estimated .2 person-hours per student for a single formal speech 
' situation (Rubin ft Basale, in preparation) • 



Conclusion » 

> 

Speech performance rating procedures represent a challenge to 
adequate measurement. They require a substantial allocation of 
institutional resources. They can, however, provide valid indicators of 
oral proficiency, and can exert positive effects on classroom 
instruction. Moreover, if one decides to test speech, there is presently 
no viable alternative to performance rating techniques. c Considering 'the 
value to students of instruction in speech communication' skills, the , 
commitment to large-scale assessment of oral proficiency via performance 
rating is, on balance, justifiable. - 
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